[EDAT24] Event-based Dataset Specialized For Manufacturing Operation Classification

3 main points

✔️ The EDAT24 dataset is a new resource for highly accurate classification of basic movements (wait, pick, place, screw tightening) in manufacturing.
✔️ It is ideal for real-time motion detection using event cameras and improves the efficiency and safety of human-robot collaboration in manufacturing.
✔️ The dataset is provided as open source and is designed to be easily accessible to researchers and scalable to include additional data.

Event-based dataset for the detection and classification of manufacturing assembly tasks
written by Laura Duarte, Pedro Neto
[Submitted on 23 May 2024]
Comments: Accepted by arXiv
Subjects:  Computer Vision and Pattern Recognition (cs.CV)

code：  

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

This paper presents EDAT24, an event-based dataset for manufacturing assembly operations. The dataset includes basic manufacturing operations (waiting, picking, placing, and screwing) and records human operators performing these operations. Data is captured using the DAVIS240C event camera, which records events as changes in light intensity occur. Event data is a lightweight data format that conveys visual information and is suitable for real-time detection and analysis of human motion. Each movement contains 100 samples, providing a total of 400 samples. Data is provided as raw data (.aedat) and preprocessed data (.npy), and Python code is also provided for researchers to add new manufacturing behaviors or extend the dataset.

Introduction

In recent years, there has been an increasing need for event-based datasets to classify and detect human motion in manufacturing. Traditional frame-based cameras produce high-quality images, but problems such as motion blur, relatively long delays, and limited dynamic range exist in high-speed applications. Event cameras are new vision sensors developed to solve these problems, operating asynchronously to provide high temporal resolution and low latency.

Event cameras are very well suited for motion detection and analysis because they record events when changes in light intensity occur. However, event-based datasets are still few and far between, especially in the manufacturing sector. This is why the authors have created a new event-based dataset, EDAT24.

The EDAT24 dataset covers basic manufacturing operations (wait, pick, place, and screw tightening) and includes 100 samples for each operation. Both raw and preprocessed data are provided, and Python code is included to allow researchers to add new behaviors and extend the dataset. Data is collected using an open source CT benchmark and captured with a static DAVIS event camera.

Figure 1 shows the frames and events of the EDAT24 data set. Green dots indicate positive polarity events and red dots indicate negative polarity events.

Figure 1: Frame and event showcase of the EDAT24 dataset taken with DAVIS240C. Positive and negative polarity events are overlaid on top of each frame as green and red points, respectively.

Related Research

Research on event-based vision sensors is progressing rapidly, with particular interest in their high temporal resolution and low latency characteristics, and Gallego et al. (2022) provided an overall overview of event-based vision and a survey of its advantages. The study confirms that event cameras outperform traditional frame-based cameras in motion detection and analysis. Specifically, they have shown improved performance in high-speed motion detection and in low-light environments.

Duarte and Neto (2023) have also proposed a method for filtering event data to classify basic manufacturing behaviors. In this study, they show that filtered event data can be used to classify manufacturing behavior with high accuracy. In particular, it is important to detect and classify manufacturing actions in real time, which is critical in a collaborative human-robot environment.

Wang et al. (2019) propose a spatiotemporal event cloud for gesture recognition and show the benefits of moving from RGB cameras to event cameras. This enables highly accurate recognition of gestures, which can help predict human behavior and ensure a safe environment.

Thus, event-based datasets and classification methods play an important role in the real-time detection and classification of human behavior in manufacturing. The EDAT24 dataset proposed in this study is expected to be an important contribution in this area.

Data Description

The EDAT24 data set contains a total of 400 sample videos. All of these videos are stored in the .aedat (AEDAT 2.0) format; the AEDAT 2.0 file format stores both frame data and event information, with each event recorded as a 32-bit wide [address, time stamp] pair. The address indicates the x-coordinate, y-coordinate, and polarity (polarity) of the event, while the timestamp is recorded in microseconds. The event acquisition rate of the data set video averages 1.65 MHz.

The directory structure of the data set is as follows

Figure 2: Directory structure of the EDAT24 dataset

Each .aedat file corresponds to a .csv file of the same name, which contains timestamps of the start and end times of the record. The timestamps of the first and last events are recorded, providing information on the exact duration of each record.

Additionally, the NumPy file format is used to store event data processed from raw data due to its fast loading time and small file size. Each .aedat file is processed individually and provided as an .npy file containing event-only data. This data includes a list of x and y coordinates for all events, a polarity list of events, and a timestamp list of events.

Each operation is for a portion of CT-Benchmark, for example, the file name pick/bridge_peg_2.aedat shows a video sequence of the second performance of the bridge peg component pick operation.

Figure 3 shows the acquisition hardware and software setup for the EDAT24 data set. the Arduino is connected to a foot pedal and two push buttons that signal the start and end of the capture system. the Python software handles the event camera and the Arduino handles communication, data recording, and all file creation.

Figure 3: Workbench setup for acquiring data from the EDAT24 dataset

Experimental design, materials and methods

Data was collected from manufacturing assembly tasks within the Coordinated Task Benchmark (CT-Benchmark). The benchmark offers a variety of assembly options, and EDAT24 includes the manufacturing actions "wait," "pick," "place," and "screw tightening." In particular, data was collected with a focus on human-robot coordinated assembly tasks.

In Figure 4, the workbench setup for data collection is shown: the DAVIS240C event camera is positioned in a fixed position and captures video of the data set. The camera is connected to a computer via a USB cable and uses the open source jAER software to capture and visualize the data. Because the event camera is fixed, stationary elements are not recorded in the event data. Therefore, the event camera only captures the movement of the human hand or torso.

Figure 4: EDAT24 dataset acquisition hardware and software setup

The DAVIS240C also simultaneously captures 20 grayscale frames per second. These grayscale frames are also included in the data set and can be accessed by decoding the .aedat file.

The hardware setup includes an Arduino connected to a foot pedal and two push buttons that signal the start and end of the capture system. By pressing the foot pedal, a signal is sent to the Arduino, allowing the user to easily start and stop recording. An additional button allows the user to continue or interrupt the recording process.

Custom-built Python software handles communication with the event camera (via jAER) and Arduino, data recording, and creation of all files in the data set. The software is freely available in the GitHub repository and includes Arduino code and a video demonstrating the use of data acquisition.

Limit

There are several limitations to the data collection methodology in this study. First, although 400 videos were recorded, the total recording time is only about 16 minutes and 40 seconds (about 2.5 seconds for each video). However, it contains 100 different samples for each basic behavior (class), which is enough data to train learning algorithms in general and deep neural networks in particular.

Another limitation of this data collection method is the fixed position of the camera. Because the event camera is fixed, stationary elements (e.g., background) are not recorded in the event data, only moving elements (e.g., hand or torso movement). This characteristic makes the dataset suitable for specific motion detection, while limiting overall environmental awareness.

In addition, the structure of the dataset is designed to facilitate the addition of new data. This allows researchers to easily expand the dataset by adding new samples.

With regard to the ethical aspects, the author himself appears in all the videos, and since only his hands and torso are shown in the videos, no specific individual can be identified. Therefore, informed consent was not required.

Conclusion

This study introduced EDAT24, an event-based dataset for classifying basic movements in manufacturing assembly operations. Event cameras provide high temporal resolution and low latency in motion detection and analysis, making them very useful in the study of human-robot collaboration in manufacturing. This dataset contains both raw and preprocessed data, and Python code is provided for researchers to add new behaviors and extend the dataset.

The EDAT24 dataset contains 100 samples for basic manufacturing operations (wait, pick, place, and screw tightening), for a total of 400 samples. The dataset is suitable for real-time detection and classification of human actions and provides a basis for improving the safety and efficiency of human-robot collaboration in manufacturing.

The dataset is open source and publicly available for easy access and use by researchers. This is expected to advance research on the detection and classification of human behavior in manufacturing, leading to safer and more efficient collaborative work environments.

• 
• 
• 

Categories related to this article

• Article
• Datasets
• Dataset
• Manufacturing
• Video