Human emotions exist on a spectrum. More than 50 individual emotions within the categories of love, surprise, joy, anger, sadness, and fear represent the spectrum of human conversation. Each of these emotions has hundreds of potential signals – some of them through body language, some through words, and others through tone of voice. The ability to recognize these signals and respond indicates someone’s emotional intelligence, which strongly correlates to their potential success in life.
The same is true for machines. As artificial intelligence becomes more ubiquitous in consumer applications, it must develop a greater understanding of human emotion and how to respond. That’s where emotion AI comes in. In many applications, more than an “emotionless robot” is needed to replace or augment a human counterpart. Emotion AI is enabling engineers to program systems that interact with human users to recognize and respond to signals of human emotion in both vocal patterns and body language. The more these systems become agile in identifying and responding to individual patterns, the more nuanced they will be in a consumer-facing role.
How AI recognizes emotion in voice
Until very recently, AI has been able to recognize and respond to only a small percentage of human communication—the quantifiable words we speak. Even then, it is often a very literal conversation partner. Voice assistants that are trained to process simple queries often fail to identify sarcasm, exasperation, or hyperbole, among other human speech habits. Emotion AI addresses this by identifying and understanding emotional prosody and intonation in voice to understand the subtext of the conversation.
Through careful development of systems that can fully understand the 50 emotional signals humans imbue in their vocal patterns, these systems can measure and track changes in speed, volume, pitch, timbre, and elongated pauses in speech. Prosody can have a direct impact on the meaning of even a few words. Combined with colloquialisms, the combination of key phrases, the clauses implemented in a conversation and even the non-linguistic sounds people make, emotion AI can piece together an entirely new map of the intention behind a conversation, going well beyond the surface level words spoken.
These systems work by collecting behavior signals related to emotions, perceived thoughts, behaviors identified in speech, ideas and beliefs. Humans are incredibly nuanced. An eye-roll can convey a tremendous amount of information in a split second. What we’re learning is that the human voice is highly indicative of these emotional shifts. That eye roll is accompanied by a quarter-sigh or a short pause in speaking. Emotion AI can identify and catalog those changes.
The application of emotion AI
As machines learn how to identify and respond to shifts in human emotion, what impact can this have on commercial industries? Already, there are dozens of industries putting the new technology to use, from augmenting human operators to providing valuable additional layers of engagement.
In the customer service industry, Allstate is using its internally developed AI agent, Amelia, to provide real-time feedback on service calls to agents. IBM’s Customer Care Voice Agent, powered by Watson, is being integrated by call centers to provide an additional layer of engagement between human agents and automated responses to customer inquiries. MetLife’s AI system analyzes calls and provides feedback and suggestions to agents on how to improve their performance in real-time. These whisper agents are a powerful resource for customer service agents who frequently must communicate with people who reach “heightened emotional states.” It’s difficult enough to maintain a professional conversation in the face of an angry, at times combative customer. It’s even more difficult for the agent to recognize when they are starting to respond in kind, mirroring those negative emotions and letting frustration seep into their voice. Whisper agent technology uses emotion AI to recognize these cues and provide real-time feedback to both the agent and management if adjustments are needed.
The healthcare industry is finding similar benefits to emotion AI. Passive tools designed to detect stress-levels in patients are being deployed in operating and exam rooms to provide real-time insights to physicians if care needs to be adjusted. Emotion AI is also being implemented to support physicians and help reduce the burden of administrative stress. With burnout a genuine concern among physicians who spend 49 percent of their time completing administrative work, and only 27 percent of their time with patients, emotion AI is being paired with voice assistant technology to offload some of the burdens. Tools are also being designed to help with clinical notes and test orders, process medical queries, and respond in kind to the stress levels of the doctor. Combined with AI systems that are already helping to improve diagnosis rates, there are ample applications for healthcare-oriented AI tools in the coming years.
In the financial industry, AI is expected to save as much as $1 trillion in the next decade, tackling the vast support costs that banks incur annually. Chatbots that utilize emotion AI are better able to engage with and respond to customer inquiries, providing an equivalent high-level experience for which human intervention can be reserved for more significant problems. With 73 percent of C-level executives citing the transformative value of AI, the systems capable of evaluating and processing human emotion will be most valuable in HR and marketing. Already, 47 percent of human resources executives when surveyed said they are already invested in AI projects for their companies that automate previously human-operated tasks like candidate screening. Emotion AI is similarly being used to evaluate and detect unconscious bias from interviewers (both in HR and management) and providing a more streamlined, responsive training and onboarding experience.
In education, emotion AI is being used to measure the response of students to assignments and in-class interactions. With the use of AI, teachers can intervene when necessary to adjust assignments for those who are struggling, provide an additional challenge for those who are further ahead, and adjust overall lesson plans.
The future of emotion AI
Applications or emotionally intelligence AI are already becoming prevalent across industries. Systems that can both detect human facial expressions and vocal cues are being used to identify and process emotional input during customer service, training, healthcare, and financial interactions, and education.
For machines to supplement or even replace humans one day for fundamental interactions, they must be capable of identifying and responding to the vast amount of data that comes in the human voice and facial expressions. Studies have shown that our sense of hearing is incredible acute at identifying emotions in conversation. We can even “hear a smile” as has been shown in several studies. People can identify emotions in a voice even when they do not speak the language of the individual with whom they are conversing. It’s an innate ability we’ve developed over millennia of social interaction, and machines are just now beginning to decode it.
There are currently more than 1 billion smart devices with voice assistants – including smartphones, dedicated voice assistants, and integrated devices in cars and appliances. AI is becoming an integral part of our lives, and voice search will be how we engage with technology as a result. Because words in a conversation are only a fraction of the underlying meaning, it has become increasingly important for machines to see under the surface, bridging the gap between perception and the content of our words, and providing support to professionals who regularly engage with the public.
While we are a long way off from entirely replacing human agents with machines, we will continue to see support tools that help improve these interactions, supplement surface level engagements, and catalog the most frequent interactions in these situations. Emotion AI will be at the core of this push in voice and other, emerging technologies.