Design and development of Huangmei Opera intelligent digital human based on multimodal interaction

Journal Information

J. Hefei Univ. Tech. (Nat. Sci.), Jul. 2026, Vol. 49 No. 7: 872-881

DOI: 10.3969/j.issn.1003-5060.2026.07.002

Authors

TAO Jianlin $ ^{1} $, HU Mengxia $ ^{2} $, CHEN Jiaojiao $ ^{2} $, CHEN Xiaohui $ ^{1} $, WANG Guangjun $ ^{2} $

(1. School of Information Technology, Anqing Vocational and Technical College, Anqing 246003, China; 2. School of Computer Science and Information Engineering, Anqing Normal University, Anqing 246011, China)

Abstract and Keywords

Abstract: Against the backdrop of cultural digitalization, aiming to address the issues of insufficient interaction in the digital communication of intangible cultural heritage (ICH) operas, inadequate fusion of multimodal information, and low level of in-depth cognition of Huangmei Opera among young groups, this study takes the living digital inheritance of Huangmei Opera as the objective, constructs a framework of Huangmei Opera intelligent digital human system based on multimodal interaction, and conducts technical verification with Feng Suzhen, a classic character from Huangmei Opera, as a case study. The system integrates three core modalities, namely speech, vision, and motion: in the speech dimension, it achieves dialect recognition and aria analysis; in the vision dimension, it integrates a high-resolution network (HRNet) to ensure the accuracy of face and facial expression recognition; in the motion dimension, it relies on optical-inertial capture and dynamic time warping (DTW) algorithm to match stylized movements such as "orchid fingers". Furthermore, it realizes multimodal information fusion based on the attention mechanism, supporting voice, gesture, and immersive interaction, and forms a technical path characterized by technology empowerment, multimodal fusion, and scenario-driven application. This study provides technical reference for the digitalization of cultural heritage and holds theoretical and application value for advancing the research on multimodal human-computer interaction.

Keywords: intelligent digital human; multimodal interaction; Huangmei Opera; speech recognition; face detection; motion capture

作者信息

陶健林 $ ^{1} $，胡孟霞 $ ^{2} $，陈姣姣 $ ^{2} $，陈晓慧 $ ^{1} $，王广军 $ ^{2} $

（安庆职业技术学院信息技术学院，安徽安庆246003；2. 安庆师范大学计算机与信息学院，安徽安庆246011）

摘要和关键词

摘要: 在文化数字化背景下, 针对非遗戏曲数字化传播交互缺失、多模态信息融合不足、黄梅戏青年群体深度认知度低的问题, 文章以实现黄梅戏活态数字化传承为目标, 构建基于多模态交互的黄梅戏智能数字人系统框架, 并以黄梅戏经典角色冯素珍为实例开展技术验证。系统融合语音、视觉、动作三大模态: 语音实现方言识别与唱腔解析, 视觉融合高分辨率网络保障人脸与表情识别精度, 动作依托光学-惯性捕捉及动态时间规整匹配“兰花指”等程式化动作; 基于注意力机制实现多模态信息融合, 支持语音、手势及沉浸式交互, 形成“技术赋能-多模态融合-场景驱动”的技术路径。该研究为文化遗产数字化提供一定的技术参考, 对推动多模态人机交互研究具有理论与应用价值。

关键词: 智能数字人；多模态交互；黄梅戏；语音识别；人脸检测；动作捕捉

Funding

安徽省高等学校科学研究资助项目（2023AH053073）；安徽省高等学校质量工程资助项目（2024cxtd275）

合肥工业大学学报(自然科学版)

Navigation