AnyGPT

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model.

Figure 1: An overview of the AnyGPT model architecture. All modalities are tokenized into discrete tokens, upon which the LLM performs multimodal understanding and generation autoregressively. Only data pre-processing and post-processing are required, with the model's architecture and training objectives remaining unaltered.

Figure 4: The construction process of the multimodal interleaved instruction dataset AnyInstruct is divided into two stages: Generation of text-based conversations incorporating multimodal elements and Text-to-Multimodality Conversion. The first stage generates topics, scenarios, and textual dialogues, while the second stage produces the final multimodal dialogues.

voice prompt (to clone):

(Transcription: write a spring poem)

(Transcription:

Spring's beauty,
The grass is green,
Colors abound,
A blooming spring we'll see,
Nature's miracle,
A new season of grace,
Spring's here, it's Spring!)

angry voice prompt1 (to clone):

(Transcription: Generate an angry dramatic character line)

(Transcription: I'm not going to take this anymore! I know it's a long-running joke but please, let me live in peace!)

(Transcription: I've been too tired recently. Where do you think suitable to relax?)

(Transcription: How about a serene lakeside?)

(Transcription: Can you give me some relaxing music?)

(Transcription: Enjoy this peaceful setting.)

(Transcription: Can you draw me a picture of sunny beach?)

(Transcription: Give me a similar style of music?)

(Transcription: Can you convert the feeling of this music into an image?)

(Transcription: The music feels energetic and triumphant.)

(Transcription: What do you see in this picture?)

(Transcription: Unity and the warmth of human connection shining brightly out here. Want to feel it too?)

(Transcription: Generate a music for it.)

Can you show me a picture of a sunny beach?

Can you give me a piece of similar style of music?.

Draw a scene of a mysterious forest where night has fallen and moonlight filters through the treetops. In the center of the image, a winding stream quietly flows, reflecting the moonlight.