[CoMat] Resolve The Discrepancy Between Text And Image

3 main points
✔️ The fundamental problem with current text-image generation models is the lack of attentional activation of text tokens.
✔️ The proposed method, CoMat, uses the ImageCaptioning model to evaluate text-image consistency and improves the Diffusion Model to significantly improve consistency.
✔️ Experimental results show that CoMat can be trained end-to-end without the need for additional data and shows significant performance improvements in quantitative and qualitative evaluation. Further improvements are expected in the future, such as through the use of multimodal LLMs.

CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
written by Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, Hongsheng Li
(Submitted on 4 Apr 2024)
Comments:Project Page: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

code：  

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

In the area of text-to-image generation, the diffusion model has been very successful in recent years. However, improving the consistency between generated images and text prompts remains a challenge.

The paper points out that underutilization of text conditions in the diffusion model is the root cause of alignment. It then proposes a new method, CoMat, which optimizes the alignment of generated images and text prompts by utilizing an image-captioning model. It also introduces a method to improve binding between attributes and entities, as well as a fidelity preservation module to preserve generative capability.

Experimental results show that the proposed method, CoMat, can produce images that are significantly more consistent with textual conditions than existing baseline models. This paper is an important contribution to the field as it presents new insights and effective methods for improving text-to-image alignment.

Related Research

In recent years, three main approaches have been proposed to improve text and image alignment.

1. Attention mechanism-based methods: Attempts to improve alignment by adjusting attention values for textual conditions; Attend-and-Excite [6] and SynGen [40] are examples of such methods.

2. Plan-based method: This approach uses a language model to generate the layout first, and then uses a diffusion model to generate the images; examples include GLIGEN [28] and RP G [59].

3. Reward optimization using image understanding models: This is the optimization of diffusion models using the output of VQA models or image capturing models for rewards; DreamSync [46] and CoMat proposed in the paper belong to this category.

Proposed method (CoMat)

CoMat is a diffusion model fine-tuning method that utilizes an image-text concept matching mechanism.

The specific flow is as follows (see above).

1. generate images from text prompts using a diffusion model.

2. input the generated images into a pre-trained image capturing model.

3. in the concept matching module,the consistency score between the text output by the captioning model and the original prompt is the optimization target of the diffusion model.

This will lower the output of the captioning model if the concept of a prompt is missing from the generated image, and the diffusion model will be induced to generate an image to include that concept.

Further,

The attribute concentration module also considers the spatial consistency of entities and their attributes.

5. the fidelity retention module introduces adversarial loss to preserve the original generation capacity.

Combining these three modules, CoMat is unique in its ability to generate high-quality images consistent with textual conditions.

Experiment

The main experimental setup is as follows

- For the base model, we primarily used SDXL [36]
- For the image capturing model, we used BLIP [25]
- For the training data, we used a total of about 20,000 text prompts from T2I-CompBench [21], HRS-Bench [3] and ABC-6K [15]First, Table 1 shows the quantitative evaluation results using T2I-CompBench.
- CoMat-SDXL significantly outperforms the baseline in the attribute binding, object-relationship, and complex composition categories.
- The improvement in attribute binding is particularly significant, with a significant improvement of 0.1895 points.

Next, the TIFA benchmark evaluation results are shown in Table 2.
- CoMat-SDXL achieves the highest TIFA evaluation score as well, improving by 1.8 points.

In addition, Figure 6 visualizes experimental results that demonstrate the importance of the fidelity preservation module.
- It can be seen that without the module, the quality of the generated image is significantly degraded.

These results confirm that the proposed method CoMat can significantly improve the alignment of text and images while maintaining the ability to generate them.

Conclusion

In this paper, we pointed out that underutilization of text conditions in the diffusion model is the root cause of alignment problems between text and generated images. We then proposed the CoMat method, which utilizes an image-captioning model and also introduces mechanisms to improve binding between attributes and entities and maintain generative capability. Experimental results showed that CoMat can generate images that are significantly more consistent with textual conditions than the baseline model. This research can be evaluated as providing new insights into the text and image alignment problem and proposing an effective solution.

The proposed method CoMat has the advantage of being an end-to-end fine tuning method and can be used in combination with other methods. In the future, CoMat's performance may be further improved by utilizing large-scale multimodal LLMs. In addition, we expect to develop a wider range of applications, including application to 3D domains. Alignment of text and images is an important issue, and it is hoped that the results of this paper will help to expand the range of applications of diffusion models.

• 
• 
• 

Categories related to this article

• Computer Vision
• Pattern Recognition
• Computation And Language