The Research team led by Prof. XUE Wei, Assistant Professor of the Division of Arts and Machine Creativity (AMC) and Division of Emerging Interdisciplinary Areas (EMIA), developed AudioX, an innovative model that generates high-quality audio and music from diverse inputs including text, video, and images.

With concerted effort by TIAN Zeyue, JIN Yizhu, and YUAN Ruibin, PhD students of the Individualized Interdisciplinary Program (IIP) and other members at HKUST, the team developed a system using a diffusion transformer architecture with a multi-modal masking strategy. This system creates a unified representation space across different types of data, allowing the single model to establish associations between various modalities, similar to how the human brain integrates sensory information.

AudioX supports multiple tasks within one framework, from text-to-audio conversion to music completion. The technology could transform creative industries, allowing filmmakers to automatically generate sound effects from visual footage, content creators to add contextually appropriate music to videos, and game developers to create dynamic audio environments.

The research team is now working to extend the model to long-form audio generation and integrate human aesthetic preferences through reinforcement learning.

Click to read the research paper: https://arxiv.org/abs/2503.10522

News Coverage
Tech Xplore– New model can generate audio and music tracks from diverse data inputs
https://techxplore.com/news/2025-04-generate-audio-music-tracks-diverse.html