Multimodal Emotion Detection with Sentimental Analysis Chatbot

DOI

No DOI

Published In

Volume 5, Issue 3 (2026)

Published Online

September-December 2026

Pages

683-690

Article Type

Original Article

Check for updates

No DOI available for cross-reference

Recognizing emotions is a vital part of human intelligence in interaction with computers, yet most traditional single-modality systems, which depend only on text, speech or facial data, cannot accurately capture the complexity of human emotions. This research proposes an approach to emotion detection that uses text, voice and face simultaneously to provide comprehensive and contextually aware affect analysis. The system combines three modality pipelines: Natural Language Processing with NLTK preprocessing, VADER scoring and a fine-tuned BERT emotion model; speech emotion recognition with Librosa-based feature extraction, MFCC computation and an SVM classifier trained on the RAVDESS dataset; and facial emotion recognition with a convolutional neural network trained on the FER2013 dataset. A reliability-aware weighted fusion procedure based on modality confidence scores achieves an accuracy of 91.3% and an F1-score of 0.91. The system also computes an Emotional Incongruence Index for inconsistency detection and uses emotion causality modeling to find contextual triggers of emotions. Finally, the detected emotion conditions a GPT-2 medium chatbot for empathic dialogue.