OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
Jun Zhan*, Chen Yang*, Yitian Gong*, Donghua Yu, Kuangwei Chen, Wenbo Zhang, Kexin Huang, Qi Luo, Zhe Xu, Ying Zhu, Jin Wang, Tengyue Zhang, Qi Chen, Cheng Chang, Songlin Wang, Junqi Dai, Jiasheng Ye, Xiaogui Yang, Tianyi Liang, Xiangyu Peng, Zhaoye Fei, Shimin Li, Qinyuan Cheng, Xie Chen, Xinchi Chen, Xipeng Qiu
Project

• Role: Project Lead
• OmniVAE is a jointly trained audio-video VAE that learns fine-grained cross-modal alignment, improving generation quality and audio-video synchronization without additional inference cost.

VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions
Jun Zhan*, Mingyang Han*, Yuxuan Xie*, Chen Wang, Dong Zhang, Kexin Huang, Haoxiang Shi, DongXiao Wang, Tengtao Song, Qinyuan Cheng, Shimin Li, Jun Song, Xipeng Qiu, Bo Zheng
Project

• Role: Project Lead
• VStyle is a benchmark for voice style adaptation in speech language models, with automated evaluation via LALM.

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Jun Zhan*, Junqi Dai*, Jiasheng*, Yunhua Zhou*, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang1, Xipeng Qiu

Project

• Role: Project Lead
• AnyGPT is the first any-to-any multimodal LLM based on discrete representations.
• Academic Impact: This work has received 800+ stars on GitHub, promoted by more than 10 media and forums, such as 机器之心

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu

Project

• SpeechGPT is the first LLM with intrinsic cross-modal generation abilities.
• Academic Impact: This work has received 1.3k+ stars on GitHub, promoted by more than 10 media and forums, such as 机器之心

MOSS: A Conversational Language Model

• Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, Xipeng Qiu.
• 🌟 MOSS is the first ChatGPT-like LLM in China, and fully open sourced with 12k+ stars. GitHub

SpeechGPT 2.0-preview: End-to-End Human-Like Spoken Chatbot

• Dong Zhang, Qian Tu*, Ruifan Deng*, Jun Zhan*, Zixin Wang*, Xingjian Zhao, Ke Chen, Xin Zhang, Pengyu Wang, Zhaowei Li, Shimin Li, Yaqian Zhou, Xipeng Qiu.
• 🌟 An end-to-end spoken dialogue LLM that directly models audio and supports expressive conversations across emotions, speaking styles, and voices, together with natural real-time interaction.
• Project · Code