Runner-up of CHI 2019 Student Design Competition
(Top 2 out of 93 teams)
Beijia Wang, Mengxiao Song, Yuting Han
Oct, 2018 - May, 2019
VUI user flow
web mid-fi prototype
all video productions
information card design
participated in research
How can we bridge the communication gap between the hearing and hard-of-hearing communities?
Being a prevalent disability in the United States, many people have little understanding of hearing loss. As the existing products mainly aim at improving the ability of having a better communication from only one side: people with hearing loss, we recognise communication as a two-way thing. So we face the challenge to make the other side: hearing people, to be more aware of how to have a successful conversation with hard-of-hearing people.
How HearU works
HearU is an integrated system that includes an interactive VUI display, a companion website, and physical information cards. It targets people who have little understanding of hearing loss, and designed to raise their awareness towards it.
featured interaction 1
When people walk by the display, use prompts to draw attention and start the interaction.
featured interaction 2
The display records and plays back the processed sound as well as showing incomplete sentence to create the hearing loss experience.
featured interaction 3
After the initial conversation, the display reinforces the interaction by providing communication tips and allowing them to interact with it again.
HOW WE GOT THERE
The process to design HearU comprised of three major phases: 1. research, 2. ideation, 3. implementation & iteration. We spent our first three months to gain understanding about the problem space and insights from people who are both hard-of-hearing and hearing; then we quickly generated concepts and built prototype for our initial design; then in the rest three months, we tested our design with both hard-of hearing and hearing people to validate the concept and usability, along the way, we also implemented the functional prototype and kept iterating the design based on feedback from testings.
Understand the Problem Space
We looked at more than 20 publications about hearing loss.
Hearing loss is a major disability in the United States.
Hearing loss is one of the most prevalent public health crises given its significant physical and psychological repercussions on adults.
Hard-of-hearing people experience emotional frustration and social isolation due to its invisibility.
Hearing loss is not immediately noticeable, and it influences self-disclosure, social-function, relationships, and day-to-day functioning.
We conducted several interview sessions with an audiologist to further understand what is hearing loss and how hearing loss affects people.
Hearing loss frustrates people from both hearing and hard-of-hearing communities.
To our surprise, hearing loss not only frustrates hard-of-hearing people but also hearing people when they fail to have successful communication.
To understand the current state of hearing loss and existing solutions to communication barriers, we conducted a competitive analysis. Most existing products focus on using technology to help hard-of-hearing people’s hearing, for example, by providing hearing aids to amplify sounds. However, these products are not always accurate or effective enough.
Most existing products focus on hard-of-hearing people while many are not effective enough.
Get insights from both hard-of-hearing people and hearing people
From our initial research, we had the hypothesis that hearing people do not have enough understanding towards hearing loss. So we posted survey to collect response from general public.
Hearing people know little about hearing loss.
Only 10% of participants have the understanding of how to facilitate effective communication-by speaking slower and clearer, or facing people directly during the conversation.
Two Focus Group Sessions
We conducted two interview sessions with both hard-of-hearing people (six in total) and their communication partners (four in total) in order to understand their communication and experience from both perspectives.
There is a huge communication gap between hearing people and hard-of-hearing people.
This insight is aligned with our initial finding that hearing loss has an impact on both hearing people and hard-of-hearing people when unsuccessful communication frustrates them. We found that is also because that hearing people cannot fully understand the experience of hard-of-hearing people.
Hard-of-hearing people tend to take more responsibility in the conversation.
People from both communities are unsure about how to communicate around hearing loss. Hard-of-hearing people tend to take more responsibility in the conversation while there is also room for improvement that can be rendered possible by the hearing side.
Based on insights, we came up with three design principles to inform our design.
Motivate people to learn more about hearing loss
The design response should motivate people to care about hearing loss.
Contribute to effective communication
The design response should bridge the communication gap by contributing to more effective conversation.
Bring uplifting and engaging experience
The overall experience of design should be uplifting and engaging.
Explore design opportunities
With the insights from research, we brainstorm 30 ideas.
Then we narrowed down to three: Fill in Blanks (participatory design), Happy Dining (infrastructure design), and Speak to Me (communication device). We were able to evaluate these three concepts and further down-selected to one based on design principles.
Fill in Blanks
Speak to Me
Featured Iteration 1
Make the website more accessible
To test our design concept, we created a video prototype and conducted a Wizard of Oz study. We started each test by playing the “fill in the blanks” task video while observing participants’ interactions. After the participant spoke out the required sentence, we played the pre-prepared audio recording.
Proper visual feedback is essential for vocal interaction.
All participants felt the real-time voice visualisation was really helpful, they were able to understand it was a voice-based user interface.
More incentives are needed to make the transition from display to website.
The website address alone at the end of the task screen cannot motivate participants to take out their phone and check out the website, the website should be more accessible.
Iteration - Information Card
As a result, physical information cards with the website address were added to the HearU integrated system. These cards are printed in standard business size and placed together with the interactive display allowing people to check for more information from the website with convenience.
FEATURED ITERATION 2
Bring inclusive and conversational experience
After we made the changes, we implemented the functional prototype. We built a web-based application using voice recognition library to handle voice input, and voice processing library to filter out the high-frequency sounds to simulate the experience of hard-of-hearing. Since the library doesn’t work the same way as natural language processing, in order to reduce the potential recognition failures, instead of providing multiple prompts for users to interact with, we chose the most feasible one as the given prompt.
Concept Validation with Hard-of-hearing People
To further make sure we provided experience that did simulate the experience of hearing loss, we demonstrated the whole interaction to hard-of-hearing participants. From their feedback, they validated the accuracy of the whole experience, however, there were terms that needed to be tweaked to make the design more inclusive.
Testing with Experts
We tested our functional prototype with an audiologist, a professor specialised in hearing loss, and a representative from Hearing Loss Association of America in order to make sure we represented the accurate experience of hard-of-hearing people.
Conversational interaction makes the experience more engaging
Testing with Hearing People
From the testings with hearing people, we learned that giving user control as well as providing the interaction that simulates a natural conversation will greatly improve the experience. Also, some of them hoped to share this experience to their closed ones or they wanted to try again after the first try.
Make the experience sharable and reinforce it
Iteration - Replace Terms that May Cause Misunderstanding
- We replaced all “You” and “They” on website with hearing people and people with hearing loss.
- Instead of using “Christmas”, we changed it to “Birthday” to bring a more universal experience.
Iteration - Make Instructions Clearer for Each Step
- More instructions were added to the flow, and highlighted those steps that required an action from users.
- Give more user control for key steps, let them get prepared for what is coming.
Iteration - Reinforce the Experience on Website
- The same vocal interaction experience were added to the landing page of the website.
- Host the functional prototype with server in stead of local host to make a web-based, sharable link.
Revised Flow - Interactive Display
Revised Flow - Interactive Display
HearU as a bridge to weave the hearing and hard-of-hearing communities
We tested HI-FI prototypes with 33 participants including six hard of hearing people and three experts. Our design received 4.7 average accuracy rating out of 5 for simulating the hearing loss experience. All of our hearing participants expressed that they are more aware of how the way they speak would affect other people and would want to change their speech pattern in the future for more effective communication. Throughout our entire testing sessions, the majority of our participants found this experience interesting and uplifting.
Make a bigger impact
Based on the positive feedback we received, we are working with the Hearing Loss Association of America on future implementation and how to bring bigger impacts to society. We believe our design could make a difference and weave the hearing and hard of hearing community together.
Thank you for reading!