T2I-Adapter: Frontiers In Text-to-Image Conversion Technology

3 main points
✔️ T2I adapters aim to improve control by leveraging tacit knowledge about generation.
✔️ The low-cost adapter model provides lightweight and effective control by learning the conditional information and consistency of the T2I model, rather than learning new features.
✔️ The proposed T2I adapters excel in both generative quality and controllability, and future research is expected to advance multimodal control methods.

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
written by Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie
(Submitted on 16 Feb 2023 (v1), last revised 20 Mar 2023 (this version, v2))
Comments: Tech Report. GitHub: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)

code：  

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

The paper focuses on a large text-to-image model, noting its exceptional generative capabilities, but also the difficulty the model has in accepting precise instructions. Specifically, a method is proposed that uses features implicitly learned by the model to control the generation process in greater detail.

The proposed method introduces simple, lightweight adapters that allow the model to leverage the knowledge it has learned internally for external instructions, while leaving the larger model intact. This allows multiple adapters to be trained to respond to different conditions, allowing detailed control over the color, structure, etc. of the generated images.

Finally, the proposed adapter is very easy to use and exhibits attractive characteristics in a variety of situations. Numerous experiments have shown the adapter's ability to produce excellent images. Simply put, this paper presents a method of incorporating an adapter that allows for more detailed instructions in a text-to-image model.

The figure above relates to the proposed T2I adapter. This adapter is a simple, small-scale model that provides additional guidance for the original T2I model, with little impact on network topology or generation capabilities. The T2I adapter can be used to generate more imaginative results that would be difficult with the original T2I model. A variety of guidance can be leveraged, such as color, depth, sketching, semantic segmentation, key poses, etc., which allows for local editing and configurable guidance.

Introduction

This paper discusses a model for generating images from text (the T2I model). By training this model with large amounts of data and computational power, it is now able to generate high-quality images based on specified text and prompts. The generated images contain detailed information such as textures and edges, and can also represent meaningful content.

However, the generated results are dependent on specific instructions or prompts, making them difficult for the average user to control. When used by the general public, the generated results are unpredictable and difficult to control. The proposed method attempts to take the information that the model has implicitly learned and use it to allow more specific control over the generation process.

To this end, we are trying to improve the generated results by introducing a small adapter model and adjusting the knowledge in the model and the control signals from the outside. The proposed adapter acts as an additional network, does not affect the topology of the original model or the existing model, and is simple, small, flexible and easy to use.

Using this method, different adapters can be trained to respond to different conditions, improving control over the generated results. This makes it easy for the average user to use and ensures that the generated results are predictable and stable. The proposed adapters are said to provide effective and flexible control capabilities and have shown promising results in a wide range of experiments.

The figure above relates to a simple T2I adapter, highlighting the following features: the T2I adapter does not affect the original network topology or generation capacity, is easily deployed, is a small model with about 77 million parameters and 300 million storage, and can be operated efficiently It is It is flexible, offering multiple adapters to accommodate different control conditions, which can be combined to control multiple conditions simultaneously for a wide variety of generation. In addition, they can be easily integrated into custom models and have general characteristics that can be used in a variety of situations. This demonstrates that the T2I adapter is simple yet functional, flexible, and has practical properties.

Related Research

This section introduces several methods and models for generating images. First of all, a method called Generative Adversarial Networks (GAN) is introduced and discusses how to generate great images from random data. This method is widely used for image generation, and several other methods exist, which are also mentioned.

It then focuses on conditional image generation, introducing methods in which text and other images are incorporated as conditions. In particular, it focuses on the task of using text to generate images (T2I generation) and mentions a technique that has received a lot of attention. A technique called the diffusion model is presented, which has recently been used successfully in image generation.

However, it also highlights the problem that text alone does not provide sufficient information for image generation. Therefore, the T2I adapter is introduced as a new idea. Adapters are positioned as a low-cost method of providing structural guidance for large models. This is beneficial as a more efficient method for fine-tuning models.

T2I-Adapter

This method aims to gain more control over the generation of images from text. To this end, the diffusion model, which has been the focus of much attention recently, is introduced.

This diffusion model consists of two steps. In the first step, the image is transformed into a special space and learned so that it can be restored. Next, a modified denoiser is used to remove noise in that space. This produces clean latent features and the final image.

In addition, a text-based condition section is introduced. In other words, guidance from the text is given to the generated image. However, it is sometimes difficult to provide sufficient control with text alone, and the T2I adapter is proposed to solve this problem.

The T2I adapter is a simple, lightweight feature designed to support multiple conditions. This allows control of image generation using a variety of conditions, including text as well as sketch and color information.

Finally, the optimization process is also discussed. This involves modifying the SD parameters during training to optimize the T2I adapter, with the original images, conditions, and text used for the training samples.

The overall architecture consists of two main components. The first is a stable diffusion model pre-trained with fixed parameters. And the second contains multiple T2I adapters that have been trained to adjust the T2I model's internal knowledge and external control signals. These adapters are constructed by directly adding adjustable weights ω. The detailed architecture of the T2I adapters is shown in the lower right corner.

In complex scenarios, SDs cannot generate accurate results according to the text. In contrast, our T2I adapter can provide structural guidance to SD and generate valid results.

Divide the DDIM inference sampling evenly into three phases: early, intermediate, and late phases. Observe the results of adding guidance at these stages. Clearly, the later the iterations, the smaller the guiding effect.

The effect of tertiary sampling during training is shown. Uniform time step sampling provides weak guidance, especially in color control, but a tertiary sampling strategy can correct this weakness.

Experiment

In the experiment, 10 "epochs" (iterations of learning) were performed with a "batch size of 8", where 8 images were trained together. We used a value of "1 x 10^(-5)" for the learning rate and used an optimization algorithm called Adam. This learning process was so efficient that it could be completed in less than three days using four NVIDIA Tesla GPUs.

In our experiments, we attempted to generate images using different conditions. For example, images were generated using conditions such as sketching and semantic segmentation. This gave specific guidance to the generated images and produced more controllable results. The results confirmed that the authors' method was clearer and more similar to the original image than other state-of-the-art methods.

Experiments used FID (a measure of the difference between generated and actual images) and CLIP scores (a measure of the association between generated images and text) to quantitatively evaluate the quality of the generated images, and confirmed that the authors' method showed promising performance.

It was also shown that the method can be used not only with a single adapter, but also with multiple adapters, which can be combined to accomplish a wide variety of image generation tasks. The method has the flexibility to be used with different models and newer versions simply by adding adapters to the trained model.

Finally, it was verified that the method provides high control capabilities even on small GPUs, showing that effective control can be achieved while reducing model complexity. This has led to the development of a general-purpose method that can be used in a wider range of applications.

A visualization of the comparison between the authors' method and other methods (SPADE, OASIS, PITI, SD) is provided. Clearly, the results show that the authors' method is superior to other methods in both alignment and quality of generation.

A single adapter control visualization is provided. Using the T2I adapter proposed by the authors, SD models can produce high-quality images conditioned on color maps, sketches, depth maps, semantic segmentation maps, depth, and key poses.

The image editing capabilities of the sketch adapter are visualized. At the same time, the restoration results of the SD model are shown for comparison.

The composable controls of the adapter are visualized. Specifically, the first line shows depth + key pose and the second line shows sketch + color map.

The generalizable features of T2IAdapter are visualized. The sketch adapter was trained on SD-V1.4 and can be properly executed on SD-V1.5 and custom models (e.g., Anything-V4.0).

The generation quality of the basic, small, and miniature versions of the T2I adapter are compared. All of these are shown to be attractive in terms of both generation quality and control capability.

Conclusion

The goal of this study is to explicitly exploit the capabilities implicitly learned by T2I adapters to more accurately control their generation. Rather than learning new capabilities, the low-cost adapter model learns to match the pre-trained T2I model's conditional information with its internal knowledge to achieve effective control. The simple, lightweight structure of the T2I adapter does not affect the pre-trained T2I model's ability to generate and can be widely applied to spatial color control and fine structure control, allowing multiple adapters to be easily configured for multi-condition control. Additionally, once trained, T2I adapters can be used directly on custom models as long as they are fine-tuned from the same T2I model.

As a result, the proposed T2I adapter achieves excellent control and promising generation quality, and extensive experiments have demonstrated its effectiveness. However, the multi-adapter control has the limitation that the combination of guidance functions must be adjusted manually. Future work is expected to consider adaptive fusion of multimodal guidance information and evolve toward the development of more efficient and flexible control methods.

• 
• 
• 

Categories related to this article

• Computer Vision
• Pattern Recognition