For the second year in a row, the Athena team at UC Santa Cruz qualified for the final of the Amazon Alexa Prize Socialbot Grand Challenge, an academic competition designed to advance the field of conversational artificial intelligence (AI). ). Although hundreds of teams from around the world signed up to participate, Athena was one of only nine selected to take part in this year’s challenge and is one of five to advance to the final.
Supervised by educational advisor Marilyn Walker, professor in the Department of Computer Science and Engineering (CSE) and director of the department’s new master’s program in natural language processing (NLP), the team of 11 members includes a mix of students. undergraduate and graduate studies of the CSE. , including five members of the inaugural NLP class. For these five, being part of the Athena team allows them to participate in the Grand Challenge Socialbot of the Alexa Prize while fulfilling the requirements of the NLP Program Synthesis Project.
âThe Synthesis Project is an opportunity to conduct a substantial three-quarter team research project to develop practical skills in the area of âânatural language processing,â said Walker, who mentored the team during the four years she participated, including when the team was known as SlugBot in 2017 and 2018. âThe Alexa Socialbot Grand Challenge gives NLP students the opportunity to try out their research ideas in a live system. real time that interacts in conversation with thousands of human users. “
For the other six members of the Athena team – a mix of undergraduate and graduate students in other CSE degree programs who work in Walker’s Natural Language and Dialogue Systems Lab – participate at the Socialbot Grand Challenge offers an opportunity to gain valuable experience in solving real world problems. in the fields of natural language processing and conversational AI.
The Socialbot Grand Challenge tasks teams to create a socialbot capable of leading cohesive and engaging 20-minute conversations on various current events and popular topics with users of Alexa-enabled devices. Amazon provides each team with a $ 250,000 research grant, Alexa-enabled devices, and free Amazon Web Services (AWS) cloud computing services to support their research and development efforts.
The contest started on November 9, 2020, when the socialbots were published in the Alexa ecosystem. A socialbot is randomly chosen to be tested when a user of an Alexa-enabled device says, “Alexa, let’s chat.” Once the conversation is over, the user has the option to give a verbal assessment of the socialbot’s performance and provide feedback, which is sent anonymously to the socialbot team. The team also receives a transcript of the conversation.
âYou can tell a lot from conversations with users, especially the things that aren’t working,â Walker said.
The synthesis project for the five NLP students of the Athena team aims to improve conversational AI by improving understanding of a key facet of human language that technology has historically had problems with.
âOriginally, the socialbot did not have the capabilities to understand different pronouns, so the NLP Synthesis Team worked on their own wrapped project to develop new approaches to address this issue,â Walker said.
Through the use of coreference solving algorithms – the task of finding all expressions that refer to the same entity in a text – the NLP Synthesis team is developing a conversational model capable of capturing the context of pronouns . This approach aims to solve a language comprehension problem with socialbots, which may seem trivial to humans, but has proven to be a major stumbling block for conversational AI.
“We noticed that if we don’t provide context for the different pronouns, our system won’t necessarily know how to continue the conversation, so we might lose a potential positive interaction by responding with information unrelated to what the user is talking about,” NLP student Angela Ramirez said. âThis led us to focus on using coreference resolution to resolve ambiguous pronouns that exist in user utterances. “
Ramirez and Cecilia Li, a member of the NLP team, both worked on the Capstone project and the Athena system. They often work closely together to examine marginal cases (major issues that arise in the system that impact the user experience) and annotate the data, if necessary, to improve the socialbot’s ability to understand the context of pronouns. in marginal cases.
âI’ve learned that interacting with users in a live system is generally much less predictable than exercises in textbooks,â Li said. âWith Alexa Prize being an open project, we not only need to establish the problem. on the fly and choose the right tools to solve it, but we also have to design the process every step of the way. Understanding and learning NLP in this unique environment is truly an experience that will benefit us in our future careers.
The five NLP students have always worked in tandem with the other six members of the Athena team, dividing up tasks, achieving weekly goals, and finding innovative ways to improve the socialbot so that they have a high chance of getting through. rank among the top three this year. challenge.
The winners of the Amazon Alexa Socialbot Grand Challenge will be announced in August 2021. Cash prizes will be awarded to the top three teams. For more information on the challenge, visit the Amazon Alexa Prize page.
This year’s Athena team members are Angela Ramirez, Cecilia Li, Eduardo Zamora, Phillip Lee and Jeshwanth Bheemanpally from the new NLP Masters program; and Omkar Patil (Team Leader), Davan Harrison, Juraj Juraska, Lena Reed, Kevin Bowden, and Rohan Pandey from other CSE undergraduate and graduate programs.