ARCHIVES
Original Article
Deep Learning-Based Multimodal Emotion Recognition Using Facial Expressions and Physiological Signals with Dynamic Fusion for Real-Time Human–Computer Interaction
Shinde Akshata G1
Shinde S.G2
1 PG Scholar, TPCT’s College of Engineering, Dharashiv, Maharashtra, India. 2 Associate Professor, TPCT’s College of Engineering, Dharashiv, Maharashtra, India.
Published Online: July-August 2026
Pages: 198-207
Cite this article
↗ https://www.doi.org/10.59256/ijrtmr.20260604023References
1. Vaz, Pedro J., João MF Rodrigues, and Pedro JS Cardoso. "Affective Computing Emotional Body Gesture Recognition: Evolution
and the Cream of the Crop." IEEE Access (2025).https://doi.org/ 10.1109/ACCESS.2025.3630563
2. Hoang, M. L. (2025). A comprehensive review of machine learning, and deep learning in wearable iot devices. Ieee Access.
https://doi.org/10.1109/ACCESS.2025.3573937
3. Kakade, S. (2025). A novel four-element button mushroom MIMO antenna for enhanced Sub-6 GHz 5G communication.
International Research Journal of Multidisciplinary Scope, 6(2), 397–409. https://doi.org/10.47857/irjms.2025.v06i02.03051
4. Li, S., Deng, W., & Du, J. (2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the
wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852-
2861).https://doi.org/10.1109/TIP.2021.3131812
5. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 770–778. https://doi.org/10.1109/CVPR.2016.90
6. Goodfellow, I., Erhan, D., Carrier, P. L., et al. (2013). Challenges in representation learning: A report on three machine learning
contests. Neural Networks, 64, 59–63. https://doi.org/10.1016/j.neunet.2014.09.005
7. Martinez, B., Valstar, M. F., Jiang, B., & Pantic, M. (2017). Automatic analysis of facial actions: A survey. IEEE transactions on
affective computing, 10(3), 325-347.https://doi.org/10.1109/TAFFC.2017.2731763
8. Corneanu, C. A., Simón, M. O., Cohn, J. F., & Guerrero, S. E. (2016). Survey on rgb, 3d, thermal, and multimodal approaches for
facial expression recognition: History, trends, and affect-related applications. IEEE transactions on pattern analysis and machine
intelligence, 38(8), 1548-1568. https://doi.org/10.1109/TPAMI.2016.2515606
9. Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., ... & Patras, I. (2011). Deap: A database for emotion
analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18-31. https://doi.org/10.1109/T-
AFFC.2011.15
10. Poh, M. Z., Swenson, N. C., & Picard, R. W. (2010). A wearable sensor for unobtrusive, long-term assessment of electrodermal
activity. IEEE transactions on Biomedical engineering, 57(5), 1243-1252. https://doi.org/10.1109/TBME.2009.2038487
11. Ribeiro, G., Postolache, O., & Martín, F. F. (2023). A new intelligent approach for automatic stress level assessment based on multiple
physiological parameters monitoring. IEEE Transactions on Instrumentation and Measurement, 73, 1-14.
https://doi.org/10.1109/TIM.2023.3342218
12. D'mello, S. K., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM computing surveys
(CSUR), 47(3), 1-36. https://doi.org/10.1145/2682899
13. Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S. F., & Pantic, M. (2017). A survey of multimodal sentiment analysis.
Image and vision computing, 65, 3-14. https://doi.org/10.1016/j.imavis.2017.08.003
14. Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023). A survey of deep learning-based multimodal emotion recognition: Speech,
text, and face. Entropy, 25(10), 1440. https://doi.org/10.3390/e25101440
15. Sahu, G., & Vechtomova, O. (2021, April). Adaptive fusion techniques for multimodal data. In Proceedings of the 16th conference of
the European chapter of the Association for Computational Linguistics: Main Volume (pp. 3156-3166).
https://doi.org/10.18653/v1/2021.eacl-main.275
16. Minaee, S., Minaei, M., & Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional
network. Sensors, 21(9), 3046. https://doi.org/10.3390/s21093046
17. Sreehari, P., Raghavendra, U., & Gudigar, A. (2025). A Review of Deep Learning Techniques for EEG-Based Emotion Recognition:
Models, Methods, and Datasets. F1000Research, 14, 1276.https://doi.org/10.12688/f1000research.171170.2
18. Cui, Z., Song, T., Wang, Y., & Ji, Q. (2020). Knowledge augmented deep neural networks for joint facial expression and action unit
recognition. Advances in Neural Information Processing Systems, 33, 14338-14349.https://doi.org/10.1109/TAFFC.2022.3145939
19. Li, H., Wang, N., Yang, X., Wang, X., & Gao, X. (2023). Unconstrained facial expression recognition with no-reference de-elements
learning. IEEE Transactions on Affective Computing, 15(1), 173-185. https://doi.org/10.1109/TAFFC.2023.3263886.
20. Canedo, D., & Neves, A. J. (2019). Facial expression recognition using computer vision: A systematic review. Applied Sciences,
9(21), 4678. https://doi.org/10.3390/app9214678
21. Wu, W., & Wang, W. (2023). Lidar inertial odometry based on indexed point and delayed removal strategy in highly dynamic
environments. sensors, 23(11), 5188. https://doi.org/10.3390/s23115188
22. Ahmad, Z., & Khan, N. (2022). A survey on physiological signal-based emotion recognition. Bioengineering,
9(11), 688. https://doi.org/10.3390/bioengineering9110688
23. Kakade, S., Sharma, N., & Narawade, N. (2024). High-isolation dual-band slotted patch MIMO antenna for Sub-6 GHz 5G applications.
International Journal of Advanced Technology and Engineering Exploration, 12(123).
http://dx.doi.org/10.19101/IJATEE.2024.111100612
24. Kakade, S., & Sharma, N. (2024). Design of slotted patch MIMO antenna and investigation of antenna parameters for Sub-6 5G network.
International Research Journal of Multidisciplinary Scope, 5(3), 514–523. https://doi.org/10.47857/irjms.2024.v05i03.0994
25. Kakade, S. (2024). SUB-6 5G networks: Design of button mushroom MIMO antenna and its investigation. International Journal of
Innovations & Research Analysis, 4(03-I), 21–30. https://doi.org/10.62823/IJIRA/4.3(I).6789
26. Bazargani, M., Tahmasebi, A., Yazdchi, M., & Baharlouei, Z. (2023). An emotion recognition embedded system using a lightweight
deep learning model. Journal of Medical Signals & Sensors, 13(4), 272-279.
27. Udahemuka, G., Djouani, K., & Kurien, A. M. (2024). Multimodal emotion recognition using visual, vocal and physiological signals: a
review. Applied Sciences, 14(17), 8071. https://doi.org/10.3390/app14178071.
and the Cream of the Crop." IEEE Access (2025).https://doi.org/ 10.1109/ACCESS.2025.3630563
2. Hoang, M. L. (2025). A comprehensive review of machine learning, and deep learning in wearable iot devices. Ieee Access.
https://doi.org/10.1109/ACCESS.2025.3573937
3. Kakade, S. (2025). A novel four-element button mushroom MIMO antenna for enhanced Sub-6 GHz 5G communication.
International Research Journal of Multidisciplinary Scope, 6(2), 397–409. https://doi.org/10.47857/irjms.2025.v06i02.03051
4. Li, S., Deng, W., & Du, J. (2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the
wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852-
2861).https://doi.org/10.1109/TIP.2021.3131812
5. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 770–778. https://doi.org/10.1109/CVPR.2016.90
6. Goodfellow, I., Erhan, D., Carrier, P. L., et al. (2013). Challenges in representation learning: A report on three machine learning
contests. Neural Networks, 64, 59–63. https://doi.org/10.1016/j.neunet.2014.09.005
7. Martinez, B., Valstar, M. F., Jiang, B., & Pantic, M. (2017). Automatic analysis of facial actions: A survey. IEEE transactions on
affective computing, 10(3), 325-347.https://doi.org/10.1109/TAFFC.2017.2731763
8. Corneanu, C. A., Simón, M. O., Cohn, J. F., & Guerrero, S. E. (2016). Survey on rgb, 3d, thermal, and multimodal approaches for
facial expression recognition: History, trends, and affect-related applications. IEEE transactions on pattern analysis and machine
intelligence, 38(8), 1548-1568. https://doi.org/10.1109/TPAMI.2016.2515606
9. Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., ... & Patras, I. (2011). Deap: A database for emotion
analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18-31. https://doi.org/10.1109/T-
AFFC.2011.15
10. Poh, M. Z., Swenson, N. C., & Picard, R. W. (2010). A wearable sensor for unobtrusive, long-term assessment of electrodermal
activity. IEEE transactions on Biomedical engineering, 57(5), 1243-1252. https://doi.org/10.1109/TBME.2009.2038487
11. Ribeiro, G., Postolache, O., & Martín, F. F. (2023). A new intelligent approach for automatic stress level assessment based on multiple
physiological parameters monitoring. IEEE Transactions on Instrumentation and Measurement, 73, 1-14.
https://doi.org/10.1109/TIM.2023.3342218
12. D'mello, S. K., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM computing surveys
(CSUR), 47(3), 1-36. https://doi.org/10.1145/2682899
13. Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S. F., & Pantic, M. (2017). A survey of multimodal sentiment analysis.
Image and vision computing, 65, 3-14. https://doi.org/10.1016/j.imavis.2017.08.003
14. Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023). A survey of deep learning-based multimodal emotion recognition: Speech,
text, and face. Entropy, 25(10), 1440. https://doi.org/10.3390/e25101440
15. Sahu, G., & Vechtomova, O. (2021, April). Adaptive fusion techniques for multimodal data. In Proceedings of the 16th conference of
the European chapter of the Association for Computational Linguistics: Main Volume (pp. 3156-3166).
https://doi.org/10.18653/v1/2021.eacl-main.275
16. Minaee, S., Minaei, M., & Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional
network. Sensors, 21(9), 3046. https://doi.org/10.3390/s21093046
17. Sreehari, P., Raghavendra, U., & Gudigar, A. (2025). A Review of Deep Learning Techniques for EEG-Based Emotion Recognition:
Models, Methods, and Datasets. F1000Research, 14, 1276.https://doi.org/10.12688/f1000research.171170.2
18. Cui, Z., Song, T., Wang, Y., & Ji, Q. (2020). Knowledge augmented deep neural networks for joint facial expression and action unit
recognition. Advances in Neural Information Processing Systems, 33, 14338-14349.https://doi.org/10.1109/TAFFC.2022.3145939
19. Li, H., Wang, N., Yang, X., Wang, X., & Gao, X. (2023). Unconstrained facial expression recognition with no-reference de-elements
learning. IEEE Transactions on Affective Computing, 15(1), 173-185. https://doi.org/10.1109/TAFFC.2023.3263886.
20. Canedo, D., & Neves, A. J. (2019). Facial expression recognition using computer vision: A systematic review. Applied Sciences,
9(21), 4678. https://doi.org/10.3390/app9214678
21. Wu, W., & Wang, W. (2023). Lidar inertial odometry based on indexed point and delayed removal strategy in highly dynamic
environments. sensors, 23(11), 5188. https://doi.org/10.3390/s23115188
22. Ahmad, Z., & Khan, N. (2022). A survey on physiological signal-based emotion recognition. Bioengineering,
9(11), 688. https://doi.org/10.3390/bioengineering9110688
23. Kakade, S., Sharma, N., & Narawade, N. (2024). High-isolation dual-band slotted patch MIMO antenna for Sub-6 GHz 5G applications.
International Journal of Advanced Technology and Engineering Exploration, 12(123).
http://dx.doi.org/10.19101/IJATEE.2024.111100612
24. Kakade, S., & Sharma, N. (2024). Design of slotted patch MIMO antenna and investigation of antenna parameters for Sub-6 5G network.
International Research Journal of Multidisciplinary Scope, 5(3), 514–523. https://doi.org/10.47857/irjms.2024.v05i03.0994
25. Kakade, S. (2024). SUB-6 5G networks: Design of button mushroom MIMO antenna and its investigation. International Journal of
Innovations & Research Analysis, 4(03-I), 21–30. https://doi.org/10.62823/IJIRA/4.3(I).6789
26. Bazargani, M., Tahmasebi, A., Yazdchi, M., & Baharlouei, Z. (2023). An emotion recognition embedded system using a lightweight
deep learning model. Journal of Medical Signals & Sensors, 13(4), 272-279.
27. Udahemuka, G., Djouani, K., & Kurien, A. M. (2024). Multimodal emotion recognition using visual, vocal and physiological signals: a
review. Applied Sciences, 14(17), 8071. https://doi.org/10.3390/app14178071.
Related Articles
2026
A Strategic Framework for Depth-Dependent Hydroelectric Conversion along the Indian Coastline
2026
Reimagining Development in India: A Critical Analysis of the Viksit Bharat Vision
2026
AI-Enabled Image Description: Bridging the Gap for the Visually Impaired
2026
Perceived Occupational Risks of Emergency Medical Services Personnel
2026
Origin, Growth and recent Development of Integrated Reporting (IR): A theoretical Review
2026
Smart Hostel Management System
Share Article
Or copy link
https://www.ijrtmr.com/archives/deep-learning-based-multimodal-emotion-recognition-using-facial-expressions-and-physiological-signals-with-dynamic-fusion-for-real-time-human-computer-interaction
*Instagram doesn't support direct link sharing from web. Copy the link and share it in your Instagram story or post.