Design Memorable Everydayness
HearU B.png




CHI logo.png

Runner-up of CHI 2019 Student Design Competition

(Top 2 out of 93 teams)


Beijia Wang, Mengxiao Song, Yuting Han


Oct, 2018 - May, 2019


Design Lead


concept ideation


VUI user flow

web wireframe

web mid-fi prototype

all video productions


VUI prototypes

visual design

logo design

information card design

competitive analysis

participated in research



How can we bridge the communication gap between the hearing and hard-of-hearing communities?

Being a prevalent disability in the United States, many people have little understanding of hearing loss. As the existing products mainly aim at improving the ability of having a better communication from only one side: people with hearing loss, we recognise communication as a two-way thing. So we face the challenge to make the other side: hearing people, to be more aware of how to have a successful conversation with hard-of-hearing people.



How HearU works

HearU is an integrated system that includes an interactive VUI display, a companion website, and physical information cards. It targets people who have little understanding of hearing loss, and designed to raise their awareness towards it.

how it works.png
Interactive Display

featured interaction 1

#Prompt the conversation
When people walk by the display, use prompts to draw attention and start the interaction.

featured interaction 2

#Simulation of hearing loss experience
The display records and plays back the processed sound as well as showing incomplete sentence to create the hearing loss experience.

featured interaction 3

#Call out actions towards effective communication
After the initial conversation, the display reinforces the interaction by providing communication tips and allowing them to interact with it again.

Companion Website

featured interaction

#Reinforce and extend with simulator and other resources


Design Process

The process to design HearU comprised of three major phases: 1. research, 2. ideation, 3. implementation & iteration. We spent our first three months to gain understanding about the problem space and insights from people who are both hard-of-hearing and hearing; then we quickly generated concepts and built prototype for our initial design; then in the rest three months, we tested our design with both hard-of hearing and hearing people to validate the concept and usability, along the way, we also implemented the functional prototype and kept iterating the design based on feedback from testings.



Understand the Problem Space

Secondary Research

We looked at more than 20 publications about hearing loss.


Hearing loss is a major disability in the United States.

Hearing loss is one of the most prevalent public health crises given its significant physical and psychological repercussions on adults.

Hard-of-hearing people experience emotional frustration and social isolation due to its invisibility.

Hearing loss is not immediately noticeable, and it influences self-disclosure, social-function, relationships, and day-to-day functioning.


Expert Interview

We conducted several interview sessions with an audiologist to further understand what is hearing loss and how hearing loss affects people.

expert interview-1.png

Hearing loss frustrates people from both hearing and hard-of-hearing communities.

To our surprise, hearing loss not only frustrates hard-of-hearing people but also hearing people when they fail to have successful communication.


Competitive Analysis

To understand the current state of hearing loss and existing solutions to communication barriers, we conducted a competitive analysis. Most existing products focus on using technology to help hard-of-hearing people’s hearing, for example, by providing hearing aids to amplify sounds. However, these products are not always accurate or effective enough.


Most existing products focus on hard-of-hearing people while many are not effective enough.



Get insights from both hard-of-hearing people and hearing people



From our initial research, we had the hypothesis that hearing people do not have enough understanding towards hearing loss. So we posted survey to collect response from general public.


Hearing people know little about hearing loss.

Only 10% of participants have the understanding of how to facilitate effective communication-by speaking slower and clearer, or facing people directly during the conversation.


Two Focus Group Sessions

We conducted two interview sessions with both hard-of-hearing people (six in total) and their communication partners (four in total) in order to understand their communication and experience from both perspectives.


There is a huge communication gap between hearing people and hard-of-hearing people.

This insight is aligned with our initial finding that hearing loss has an impact on both hearing people and hard-of-hearing people when unsuccessful communication frustrates them. We found that is also because that hearing people cannot fully understand the experience of hard-of-hearing people.


Hard-of-hearing people tend to take more responsibility in the conversation.

People from both communities are unsure about how to communicate around hearing loss. Hard-of-hearing people tend to take more responsibility in the conversation while there is also room for improvement that can be rendered possible by the hearing side.


Design Principles

Based on insights, we came up with three design principles to inform our design.


Motivate people to learn more about hearing loss

The design response should motivate people to care about hearing loss.


Contribute to effective communication

The design response should bridge the communication gap by contributing to more effective conversation.


Bring uplifting and engaging experience

The overall experience of design should be uplifting and engaging.



Explore design opportunities



With the insights from research, we brainstorm 30 ideas.


Then we narrowed down to three: Fill in Blanks (participatory design), Happy Dining (infrastructure design), and Speak to Me (communication device). We were able to evaluate these three concepts and further down-selected to one based on design principles.


Happy Dining


Fill in Blanks


Speak to Me


Finalised Idea



An integrated system that raises public awareness of hearing loss
Interactive Display
The “fill in the blanks” task first prompts the user to speak out the sentence shown on the display. Then the display shows the broken sentence while playing back the pre-processed voice that has been filtered with muffled sounds to match the missing letters in the sentence. To make the task more meaningful, we also embedded two tactics that can improve the conversation quality and one that can not. The display informs the user why speaking louder is the wrong way by prompting the user to do it first. After the vocal interaction, actual quotes and more information about hearing loss are shown at the end of the task to build a personal touch.

As a tool for awareness, the public display would only capture people’s attention for a short amount of time, so we decided to evolve the design from the interactive display into an integrated system. The system contains a display with a website to enable more organic interactions between the hearing and hearing loss communities. The website was designed in a one page layout to provide a brief overview of what is hearing loss, the importance of it, how to facilitate effective communication, and at the same time, allowed hard-of-hearing people to share their experience.

Featured Iteration 1

Make the website more accessible

To test our design concept, we created a video prototype and conducted a Wizard of Oz study.  We started each test by playing the “fill in the blanks” task video while observing participants’ interactions. After the participant spoke out the required sentence, we played the pre-prepared audio recording.

WoZ testing.png



Proper visual feedback is essential for vocal interaction.

All participants felt the real-time voice visualisation was really helpful, they were able to understand it was a voice-based user interface.


More incentives are needed to make the transition from display to website.

The website address alone at the end of the task screen cannot motivate participants to take out their phone and check out the website, the website should be more accessible.


Iteration - Information Card

As a result, physical information cards with the website address were added to the HearU integrated system. These cards are printed in standard business size and placed together with the interactive display allowing people to check for more information from the website with convenience.



Bring inclusive and conversational experience



After we made the changes, we implemented the functional prototype. We built a web-based application using voice recognition library to handle voice input, and  voice processing library to filter out the high-frequency sounds to simulate the experience of hard-of-hearing. Since the library doesn’t work the same way as natural language processing, in order to reduce the potential recognition failures, instead of providing multiple prompts for users to interact with, we chose the most feasible one as the given prompt.


Concept Validation with Hard-of-hearing People

To further make sure we provided experience that did simulate the experience of hearing loss, we demonstrated the whole interaction to hard-of-hearing participants. From their feedback, they validated the accuracy of the whole experience, however, there were terms that needed to be tweaked to make the design more inclusive.


Design inclusively


Testing with Experts

We tested our functional prototype with an audiologist, a professor specialised in hearing loss, and a representative from Hearing Loss Association of America in order to make sure we represented the accurate experience of hard-of-hearing people.


Conversational interaction makes the experience more engaging


Testing with Hearing People

From the testings with hearing people, we learned that giving user control as well as providing the interaction that simulates a natural conversation will greatly improve the experience. Also, some of them hoped to share this experience to their closed ones or they wanted to try again after the first try.


Make the experience sharable and reinforce it


Iteration - Replace Terms that May Cause Misunderstanding

- We replaced all “You” and “They” on website with hearing people and people with hearing loss. 

- Instead of using “Christmas”, we changed it to “Birthday” to bring a more universal experience. 


Iteration - Make Instructions Clearer for Each Step

- More instructions were added to the flow, and highlighted those steps that required an action from users. 

- Give more user control for key steps, let them get prepared for what is coming.


Iteration - Reinforce the Experience on Website

- The same vocal interaction experience were added to the landing page of the website.

- Host the functional prototype with server in stead of local host to make a web-based, sharable link.



HearU: An Integrated System
Revised Flow - Interactive Display

Revised Flow - Interactive Display


HearU as a bridge to weave the hearing and hard-of-hearing communities

We tested HI-FI prototypes with 33 participants including six hard of hearing people and three experts. Our design received 4.7 average accuracy rating out of 5 for simulating the hearing loss experience. All of our hearing participants expressed that they are more aware of how the way they speak would affect other people and would want to change their speech pattern in the future for more effective communication. Throughout our entire testing sessions, the majority of our participants found this experience interesting and uplifting.



Make a bigger impact

Based on the positive feedback we received, we are working with the Hearing Loss Association of America on future implementation and how to bring bigger impacts to society. We believe our design could make a difference and weave the hearing and hard of hearing community together.


Thank you for reading!