[OW-VISCap] Look Out For Unseen Objects - A New Approach To Understanding Open World Video

3 main points
✔️ Introduction of an open-world object query enables discovery of unknown objects without prompting, and integrated detection, segmentation, and tracking with known objects.
✔️ Applying the mask-attention mechanism to the object-to-text converter allows object-centric captions to be generated while considering the context of the entire video.
✔️ Introducing contrast loss to suppress similarity between object queries, we achieved superior performance in video understanding tasks from open world to closed world by both suppressing duplicate detection and discovering new objects.

OW-VISCap: Open-World Video Instance Segmentation and Captioning
written by Anwesa Choudhuri, Girish Chowdhary, Alexander G. Schwing
(Submitted on 4 Apr 2024)
Comments: Project page: this https URL
Subjects:  Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

code：  

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

This paper proposes an approach called OW-VISCap (Open-World Video Instance Segmentation and Captioning). The three main contributions are as follows

1. open world object query: In addition to known object queries, we introduce an open world object query to discover unknown objects. This allows for the detection of unknown objects without requiring additional input.

2. object-centered captioning with mask-attention mechanism: We have introduced a mask-attention mechanism into the object-text converter to enable it to generate descriptive object-focused captions.

3. contrast loss between object queries: Contrast loss is introduced to suppress similarity between object queries, thus reducing duplicate object detection while encouraging the discovery of new objects.

The proposed method shows excellent performance on three tasks: open-world video instance segmentation, video object captioning, and closed-world video instance segmentation. Qualitative results also show that the proposed method can detect unknown objects and generate object-centric captions.

Related Research

First, research on open-world video instance segmentation can be divided into two main categories.

1. methods that require prompts: prior knowledge of user input and ground-truth is required.

2. promptless methods: To discover novel objects, object suggestion methods and other methods are used. However, these have issues such as limited performance and inability to distinguish between open-world and closed-world objects.

On the other hand, DVOC-DS [58] is the only prior work on video object captioning. However, DVOC-DS had some issues such as its inability to handle long duration video and its inability to capture multiple actions of an object.

Other methods have been proposed to suppress similarities between object queries, such as OWVISFormer [39] and IDOL [50]. These were mainly effective in the closed-world setting, but not sufficient in the open-world setting.

Proposed Method (OW-VISCap )

First, for the open-world object query q_ow, this is obtained by encoding equally spaced grid points on the video frame with a prompt encoder (purple area on the left in Figure 2). This design allows us to prompt for the discovery of novel objects over the entire video region.

We also introduced a specially designed open-world loss function L_ow for q_ow to encourage the detection of unknown objects.

Next, for object-centered captioning, the use of a mask-attention mechanism in the object-text transformer enables the generation of captions that focus on object regions (Figure 2, right). Specifically, mask attention is applied using the object segmentation mask obtained by the detection head to generate captions that focus on local object features but also consider the overall video context.

Finally, regarding the contrast loss L_cont to suppress similarity between object queries, this has the effect of preventing overlapping detections as well as facilitating the discovery of new objects. In the closed-world setting, it helps to suppress overlapping false positives, while in the open-world setting, it helps to discover new objects.

Thus, OW-VISCap provides integrated video understanding through its unique design of open-world object discovery, object-centered captioning, and query-to-query similarity suppression.

Experiment

In this paper, OW-VISCap is evaluated for three tasks: open-world video instance segmentation (OW-VIS), video object capping (Dense VOC), and closed-world video instance segmentation (VIS).

For OW-VIS, we evaluated it on the BURST[2] data set (Tab. 1) and found a performance improvement of about 6% for objects in the unknown (uncommon) category.

For Dense VOC, we used the VidSTG[57] dataset (Tab. 2) and showed an improvement of about 7% in the correctness rate of the generated captions, although the object detection accuracy was slightly lower. This is because the proposed mask attention mechanism allows object-centric captions to be generated.

Finally, VIS was evaluated on the OVIS[36] dataset (Tab. 3) and showed comparable performance to state-of-the-art. Again, we see that the contrast loss between object queries contributes to the suppression of duplicate detection.

Fig. S1 and Fig. S2 show qualitative results for the BURST and VidSTG datasets, respectively. It can be seen that the detection and segmentation of unknown objects and the generation of object-centered captions are achieved.

Conclusion

This paper proposed OW-VISCap, an integrated treatment of video instance segmentation and captioning in an open-world setting. It features three elements: open-world object queries, mask-attention-based captioning, and contrast loss between object queries, which enable detection and description of unknown objects.

The core OW-VISCap method proposed here is also applicable to more general-purpose video understanding and has promise for real-world applications such as autonomous systems and AR/VR. Fine-grained video understanding, including unknown objects, is an important research issue, and this method can make a significant contribution to solving it.

• 
• 
• 

Categories related to this article

• Computer Vision
• Pattern Recognition
• Artificial Intelligence