Using AI to Bridge the Gap between the Hearing and Hearing Impaired

Pakhi Agarwal, DeepSign team member and 2019 Nittany AI Challenge participant.

DeepSign is the name of our team’s project, and it is something very close to me. Having a cousin who is hearing impaired, I was aware of the problems and challenges he faced daily, and that motivated me to do something to make the lives of the hearing impaired a little easier. DeepSign aims to bridge the gap between the hearing and the hearing impaired by providing them with an effective translation and communication tool.

Over the past few months, Nittany AI Alliance helped us to shape our work, after we came up with an idea to do something for The Deaf Society. We felt that being physically abled, we were all at some point unknowingly neglecting the hearing impaired. We couldn’t imagine what it would be like to live while struggling to communicate in our daily lives. How would they feel going out to public places, having a fear that something could go wrong or they could be misjudged or possibly pitied by people? We hope that DeepSign can help to make their lives easier because they can use our application when they want to communicate with others who don’t know sign language.

This project was not easy for our team. Sign language has no unique conventions and every country has its own sign language. The major problem we faced was with the dataset. There was no comprehensive dataset, so for this work we employed a signer and created our own dataset. Since it is a research-based project, we created our own model for this work. We also faced issues with compatibility, storing data, and making our algorithm work with the interface created. Since the Nittany AI Challenge focuses on user experience, we were very particular about providing the user with an easy-to-use interface.

While the work gave us a lot of challenges, it taught us how to tactfully deal with problems. And Nittany AI Alliance organizers were always there with us to help at every step and to do something meaningful for the society. This Challenge also helped us to publish research papers, one that was accepted recently and a few more that are under construction and review.

A lot of people helped shape our work with their opinions and suggestions at every step. I would like to especially thank Brad for all the guidance and support he has provided during the Challenge. He was always there for us and wholeheartedly supported us. He motivated us and pushed us to think beyond boundaries. The guidance he provided during the biweekly meetings was unparalleled. His perspective was, “Just do it and all other things will eventually come around.” 

Stay Informed

The Alliance is hard at work with AI industry partners to bring cutting-edge programming to Penn State. Don’t miss your opportunity to get involved!

Share your contact information to stay up-to-date on all of the activities, challenges, and networking opportunities the Nittany AI Alliance will be offering in the coming months. You’ll be among the first to know about AI–related programming, giving you a chance to sign up early for events that may offer limited seating or enrollment capacity.

By submitting this form, you are consenting to receive marketing emails from the Penn State Nittany AI Alliance. View our Privacy Policy.