Emotion-Sensing earphones Powered by Machine learning with IoT: For making your life better!

The Fourth Industrial Revolution is leading our society to a quick transition to the digitalization-era, deeply and inevitably impacting and altering the way humans-to-humans and humans-to-computer interaction is carried out.

IoT is the way of bringing the data in for several purposes (Leverage Machine learning models, analyze, etc.), that have a great impact on the industrial revolution by enhancing and empowering the people’s day-to-day life. At the same time, society as a whole is striving for greater life quality, and to do so health issues have to be monitored and treated in a better way. Consequently, the Internet of Things (IoT) scene is enriching itself thanks to the development of new devices in the different market segments of this industry.

Think of the situation, you are using earphones to hear the music, and also imagine what happens if the same device monitors your valence-arousal (emotion recognition) as well as your health in the backend with the help of biosignals from the body. This idea could be made possible in a highly-automated and scalable way, with the combination of IoT, wearable biosignal sensor, Artificial Intelligence (AI), and Cloud. Think of the situation, you are using earphones to hear the music, and also imagine what happens if the same device monitors your valence-arousal (emotion recognition) as well as your health in the backend with the help of biosignals from the body. This idea could be made possible in a highly-automated and scalable way, with the combination of IoT, wearable biosignal sensor, Artificial Intelligence (AI), and Cloud.

What are Wearable Sensors?

It is a device in direct contact with the human body to extract the physiological data. Wearable sensors are getting advance in the field of commercialization and medical research.

The Internet of Things, applied to the field of wearable devices, represents a disruptive combination of technologies that can allow greater customization and traceability of parameters, consequently improving people’s overall wellbeing.

Recent Developments in Printing Flexible and Wearable Sensing Electronics  for Healthcare Applications | GenesInk
Figure 1:Data collection in Wearable Sensor

Emotion sensing technologies in action with Machine Learning

There are some emotion recognition/sensing technologies already been rolled out in the real world.

Emotion-sensing using Physiological Signals: That includes wristband which has emotion recognition as one of its features. This technology which has the sensors embedded in wrist band/watch extracts various data such as heart rate (HR), blood pressure (BP), and temperature in order to define one’s emotional state. This kind of technology has a wide range of utilization, that helps in predicting any potential healthcare issues (early diagnosis) and monitor daily activities. These devices even periodically send fully analyzed reports with possible vulnerabilities/health predictions to the assigned physicians/doctors.

Emotion-sensing using speech and text: Speech-based and text-based emotion recognition is the technology which uses the complex multimodal machine learning algorithms. This technology uses a Convolutional Neural Network (CNN) and Long short-term memory (LSTM) to learn acoustic emotion features from speech signals. And Bi-LSTM (bidirectional-LSTM) used to learn emotion from text data. Then these two pipelines were applied to Dense Neural Network (DNN) to classify the emotion based on the input text and speech data.

Emotion-sensing using the facial expression: This topic was being active research in computer-vision platform, this method does not use any physiological data for emotion recognition. It mainly includes technical methods such as image processing and deep learning algorithms.

Most reliable machine learning/deep learning models used in this domain are Support vector machine (SVM), Random Forests (RF), k-Nearest Neighbors (k-NN), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU’s). As well as the combination of such machine learning techniques for example CNN-LSTM.

Breakthrough – ear-wear emotion sensing

If for many years smartwatches and smart bands have driven the demand for wearables that monitor human life, but for change, new rising technology is taking the scene. This is the case for earwear devices, commonly referred to as wearables.

According to IDC, these wearables achieved the highest year-to-year growth from 2018 to 2019, with a staggering figure of 242 percent from the previous year. In 2019, 139.4 million hearable devices were shipped, capturing 45.7 percent of 2019’s market share. This figure is expected to grow even more. Customer’s rising demand to take control of their health is highly influencing the use of wearable technology in the healthcare sector.

As a part of IoT, Hearables can offer more than connectivity amongst devices, gives the potential for new business models now and in the future. The ears represent an ideal position to retrieve the data and are metaphorically described as a person’s USB entrance. Its closeness to the brain suggests that in the future sensors which manage to retrieve this sort of data will be exploited through this type of device.

Why we do not depend on old technologies

As we have seen the research related to this field, there is some accomplishment using the psychological data and more i.e. image-based facial expression, voice, text. However, emotion-recognition using such data cannot guarantee in reliable solution. Because, emotion-sensing using face, speech and text highly depend on the expression, that widely varies with each individual and their cultural background and could be easily faked. Consider an individual in a negative state of emotion on some social occasions, he/she could relatively fake the true emotional state with a smile.

Due to such complexity in existing solutions, physiological data (heart-rate) passively measured form the human body throughout the day is used in emotion recognition which makes the system more precise in a timely manner. The sophistication of sensors embedded in-earphones(ear-wearables) helps to study brain signals (electroencephalography (EEG)) in the future.

Impact in real-world

But first, who are the beneficiaries? Humans life on earth is highly diverse and this solution can be applied to all nature of human life. To be simple, who doesn’t like music? To be particular, imagine an intellectually-disabled person. Emotion-sensing wearables can help monitor the emotional state of the person with mental and other health conditions 24/7.

Human-to-Machine interaction: The potential of IoT with emotion-sensing unlock a huge possibility in the media and entertainment industries. We can build an intimate recommendation system with the advancement in the AI and ML algorithms. AI systems provide biofeedback relating to the emotion that controls our audio/video experience in real-time.

Optimizing clinical encounter: The system equipped with the data collection protocols collaborated with IoT sensors periodically sends fully analyzed reports with possible vulnerabilities/health predictions to the assigned physicians/doctors.

Health Analysis: We analyze different parameters right at the source, making it possible to take better health decisions on identifying the right disorders, preventive measures to avoid them and to lead a better lifestyle. “We strive to permanently cure the disability, right at the source.”

Transport: The specially-abled or elderly people don’t necessarily have to visit hospitals with frequency. They could stay home and avoid excessive pain of commuting/transport while ensuring their lives are safe 24/7.

Quality of care / Emergencies: Our system identifies emergencies in real-time and sends an immediate alert to the near-by hospitals, caretakers, loved ones, or the neighbors. This ensures they get help in time even before the emergency services arrive.

Priority: We ensure their reports are fully analyzed by the caretakers/hospitals. We give them the priority in scheduling appointments, they don’t have the capability to wait long hours at the hospitals and the goal is to prioritize their safety with utmost convenience.

Future innovation with Open data: With the help of biometric data that we collect, it opens a whole new way to research/develop drugs and lays a path for new future technical innovations for human.

Conclusion: Next Evolution of earphones!

Sync the music with your mood; Make your device learn your habit; Monitor your health;

We can finally collect the raw heart rate, EEG, and motion data with the help of wearable sensors embedded in earphones. The retrieval of these metrics can be translated into more accurate and tailored information, through the implementation of artificial intelligence, in order to increase the awareness of the health and fitness conditions, early disease detection and avoidance of the potential risk in the cardiovascular system.

Furthermore, hearable devices are capable of providing voice feedback to the users, hence removing the need for users to look at a display for the information

Currently, we are working to develop emotion-sensing hearables that using the signals from the heart are capable of ‘reading’ the users emotional state. Research shows that physiological data such as the heart rate variability (HSV), can be used to determine a person’s emotional state from the valence-arousal methods. This includes understanding whether a person is stressed, happy, sad, tired, etc.

Geovisualization | planvizlab
Figure 2: Valence-arousal

By expanding the aspect of health-tracking, it could be interesting to develop a music recommendation algorithm based on Machine Learning/Deep Learning techniques and Artificial intelligence algorithms. These algorithms would get to know the user, in particular by understanding his/her music tastes and consequently understanding which kind of music are most preferable depending on the circumstance. 

Therefore, based on the users’ music taste and emotional status, it recommends the music suitable for the particular circumstance.

Moreover, embedding features such as live translation and noise-canceling could potentially add more appeal to this new product. For this reason, we are trying to understand the current state of the earphone market.

Understanding buying patterns, as well as customer’s expectations for future developments, can help the development stage of this project. So, Please give your idea about the preference for ear-wear models in your mind right now.

For additional details, contact us!

Related Articles

Leave a Comment