Review: Image Conversion Of IoT Time Series Data And Various Applications

3 main points
✔️ presents a review of these studies using image conversion/encoding techniques in the IoT domain
✔️ presents various time series to image conversion techniques
✔️ presents image conversion in various IoT applications

Image Transformation for IoT Time-Series Data: A Review
written by Duygu Altunkaya, Feyza Yildirim Okay, Suat Ozdemir
[Submitted on 21 Nov 2023]
Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Networking and Internet Architecture (cs.NI)

code： 

The images used in this article are from the paper, the introductory slides, or were created based on them.

First of all

In the Internet of Things (IoT) era, where almost all smart devices, including smartphones, embedded systems, and wireless sensors, are connected via local networks and the Internet, billions of smart devices communicate with each other, generating vast amounts of time series data. Due to the high dimensionality and high frequency of time series data in IoT, classification and regression of time series has been a challenging task in IoT. In recent years, deep learning algorithms have shown excellent performance in classifying time series data in many smart and intelligent IoT applications. However, exploring dynamic patterns and trends hidden in time series is challenging. Recent studies have shown that transforming IoT data into images improves the performance of learning models. This paper presents a review of these studies using image transformation/encoding techniques in the IoT domain. Studies are examined according to encoding technique, data type, and application area. Finally, the challenges and future potential of image transformation are highlighted.

Introduction

This study provides a comprehensive survey of the use of image conversion technologies in the field of the Internet of Things (IoT). the IoT is a network of smart devices equipped with sensors, software, and advanced technology that connect and share data with other devices and systems via the Internet IoT refers to the A wide variety of IoT applications exist, including smart homes, smart cities, smart agriculture, smart health, and smart retail.

The proliferation of IoT has generated vast amounts of time-series data, which has led to widespread time-series analysis across a wide variety of IoT domains. Traditional time series analysis methods achieve a certain level of performance through manual feature extraction and expertise, but due to the unique characteristics of IoT time series data, these methods are not always appropriate. time series data analysis of IoT devices differs from analysis of non-IoT data in its complexity presents challenges due to its complexity. Many IoT applications require real-time or near-real-time data processing, which can be technically challenging and require special infrastructure.

To address these challenges, image conversion/encoding techniques that convert time-series data into visual representations have been proposed as promising technologies. By converting data into images and applying image compression techniques such as jpeg or png, necessary information can be retained while reducing data size. Time-series data in compressed image representation can be stored or transmitted more efficiently. In recent years, researchers have focused on converting time-series data to image format due to its great success in IoT applications such as anomaly detection, fault diagnosis, and activity recognition.

This survey provides a comprehensive overview of image transformation technologies from a variety of perspectives. First, existing studies will be scrutinized based on conversion technologies, and then categorized by data type (univariate or multivariate) and application domain. So far, no survey paper on the use of image transformation techniques in the IoT domain seems to exist. To fill this gap, this paper presents a detailed analysis of the current research landscape within the IoT domain.

Motivation

In this study, the basic idea of improving the accuracy of a model is to consider changing to another model with higher accuracy. Many researchers have applied composite models, such as hybrid or pre-trained models. However, it is also important to consider whether it is possible to improve the accuracy of the model without changing the model itself. Some studies suggest that transforming time series data may be a more effective approach to improving model accuracy than changing the model itself.

There are several advantages to representing IoT data as images:

i) It is easier to visualize and analyze complex patterns and trends.

ii) Provides a visual representation of temporal data, allowing intuitive interpretation and pattern recognition.

iii) Transforming high-dimensional IoT time series data is an effective way to reduce dimensionality while maintaining temporal dependencies, which may lead to more efficient analysis and better insights.

iv) Deep learning techniques can be used to effectively analyze IoT time series data with image-based analysis in IoT applications such as pattern classification and healthcare monitoring.

These aspects form the basis of the motivation for this study. This approach has the potential to improve the accuracy of the model by changing the representation of the time series data without changing the model itself.

Introduction

Time Series Analysis in IoT

This study focuses on time series data, a series of data points collected periodically over time. Time series data are represented as X = {( _t1, _x1 ), ( _t2, _x2 ), ... , ( _tn, _xn )}, where _xi ∈ ^Rm means a vector of n time series data points and m dimensions. Time series data can be classified as univariate or multivariate.

Univariate Time Series (UTS): X is univariate if m is 1. That is, a UTS contains a single variable that is observed over time.

Multivariate Time Series (MTS): X is multivariate if m is greater than 1. In other words, an MTS has multiple variables that are observed over time.

For example, time-series data containing a city's average daily temperature is represented as UTS, while time-series data of daily weather conditions including temperature, humidity, and precipitation are represented as MTS. Although many real-world IoT systems have a variety of IoT sensors, we focus more on UTS for several reasons. First, it is difficult to get the relationships between variables correct in MTS. Furthermore, the high dimensionality of these variables makes analyzing MTS data challenging. Therefore, UTS is simpler and easier to implement than MTS. However, MTS is more complex and requires more data than UTS, but can be more accurate because it deals with relationships between different variables.

IoT time-series data is generated from a variety of areas, including remote healthcare, wearable devices, energy management, smart buildings, and transportation, and is widely used in a variety of IoT problems, including anomaly detection, monitoring systems, signal classification, fault diagnosis, and maintenance prediction (Figure 1).IoT time-series data has several unique characteristics that distinguish it from other types of data, and understanding and leveraging these characteristics is essential for effective analysis, modeling, and decision making in IoT applications. For example, high dimensionality is a key issue in IoT time series analysis that requires scalability. Also, since data is continuously generated, real-time or streaming data processing methods are needed to process data flows, perform immediate analysis, and make timely decisions. In addition, noise and missing values can degrade data quality, requiring the use of data cleaning and pre-processing techniques to ensure data integrity. Various tasks have been proposed by researchers to overcome these challenges. This paper, as well as other studies, focuses on methods as well as changes in time series data types and methods.

Image transformation

Time series image transformation, which converts time series data into a visual representation (e.g., an image), is an important process within the IoT context. This technique reduces the dimensionality of IoT data by compressing large amounts of data into a compact visual format and is more successful in extracting important features and patterns from IoT time-series data. In addition, it seamlessly integrates with deep learning algorithms such as convolutional neural networks (CNNs). These transformations enhance the analysis, interpretation, and use of time-series data in IoT applications.The process of transforming IoT time-series data into images is illustrated in Figure 2.

Figure 2: Overall framework for image transformation of IoT time-series data

Various image conversion techniques are described in the literature. These methods are generally applied directly to UTS, but usually not directly to MTS. Several fusion methods have been discussed in the literature to address this issue. Image or feature fusion is a proposed process for integrating necessary information from images or features; when converting MTS data to two-dimensional (2D) images, fusion methods can be used to combine information from different variables or data sources to create a single image representation.

One popular fusion technique in the literature is channel-based fusion, in which RGB or multispectral channel images are created by assigning each variable to a different color channel (e.g., red, green, blue). Some studies also use tensor image fusion, where MTS data are considered as tensors and patterns and interactions are extracted from the tensor data using tensor decomposition techniques (e.g., standard multiple linear decomposition). Finally, MTS can be transformed using feature level (early fusion) and decision level (late fusion). The different variables are integrated at the input stage and processed together with any methods at the feature level. On the other hand, each variable is transformed into an image separately and then these images are combined in a later stage at the decision level. Many researchers also use hybrid fusion, which performs fusion at both the decision-making level and the feature level.

Time series to image conversion technology

There are several methods to convert a one-dimensional (1D) time series into a two-dimensional image. Some of the most commonly used methods in the literature are described below. Table I also shows studies that have used these methods.

Table 1: Research on image conversion techniques for IoT.

Gramian Angle Field (GAF)

To encode time series data $X = {x_1, x_2, ... , x_N}$ is encoded into an image in three steps. This method is an effective way to convert time-series data into a visual representation and is especially useful in pattern recognition and analysis of time-series data.

Scaling: First, the time series X is scaled to the interval [0,1]. This is done according to the following equation (1)

~��=�-min(�)max(�)-min(�)

where $\tilde{X}_i$ is the i-th element of the scaled time series.

Transformation to a polar coordinate system: Next, the Cartesian coordinate system of the 1D time series is transformed to a polar coordinate system. This yields a new representation of the time series. The cosine (φ) and radius (r) of the angle are computed to represent the rescaled time series in polar coordinates using the following equation (2)

where_ti is the timestamp and N is a constant factor to normalize the range of the polar coordinate system.

Calculation of GASF and GADF: There are two forms based on the sum/difference of trigonometric functions: GASF (Gramian Angular Summation Field) and GADF (Gramian Angular Difference Field) GASF is defined by equations (3) and (4), GADF is defined by equations (5) and (6).

GASF=[cos(�1+�1)⋯cos(�1+��)cos(�2+�1)⋯cos(�2+���) )⋮⋱⋮cos(��+�1)⋯cos(��+��where I is the unit vector [1,1,. ,1].

Through the above steps, time-series data is converted into images, which can be easily handled by deep learning and other advanced analysis techniques.

Markov Transition Field (MTF)

Markov Transition Fields (MTFs) are powerful tools that preserve time-domain information within time series data and represent sequential Markov transition probabilities. By utilizing a Markov matrix of quantum bins, MTFs provide an approach to transform time series data into images.

The encoding process starts with a time series X and determines its Q quantum bins. Each xi is then assigned to the corresponding bin qj (j ∈ [1, Q]). We then obtain the adjacency matrix W = QxQ where each element wi,j represents the frequency with which a point in qj is followed by a point in qi. Finally, the Markov transition matrix is constructed as shown in equation (7) below.

Through this process, time-series data is converted into an image format that allows for a more intuitive and visual analysis of data features and patterns. This method reduces the dimensionality of the data while maintaining the temporal dependencies of the time series data, allowing the data to be represented in a format suitable for deep learning and other advanced analytical methods.

Recovery Plot (RP)

Recurrent Plot (RP) is a widely used tool for visualizing and analyzing repeated time-series behaviors generated in a dynamic framework.RP is determined by a recursive matrix by computing the pairwise distances between trajectories. Its elements are computed by the following equation (8)

where ε is the threshold value and Θ is the Heaviside function used to binarize the distance matrix, with zero value for negative arguments and unity value for positive arguments.RP reveals local correlation information and hidden patterns in time series by computing the distance matrix between subsequences RP reveals local correlation information and hidden patterns in a time series by computing a distance matrix between subsequences. This method helps to visually capture recurring patterns and dynamics within time series data and is especially useful for analyzing complex systems and nonlinear dynamics. modeling, providing important insights in time series analysis and prediction.

Short-time Fourier transform (STFT)

The short-time Fourier transform (STFT) can be viewed as a frequency domain representation of the original signal; the STFT uses a window function to extract a portion of the time domain signal and then performs a Fourier transform on it to identify various signal characteristics. The STFT for a given signal y(x) is computed by the following equation (9)

where ω(t) is the window function. Additionally, the spectrogram is generated by squaring the amplitude of the STFT:

STFT is a powerful tool for analyzing frequency components while accounting for temporal variations in signals. It allows one to capture temporal variations in signals and is particularly suited for analyzing non-stationary or time-varying signals. Spectrograms are used to visually show the temporal variation of these frequency components, helping to understand the dynamics of the signal. This method is widely used in many fields, including speech analysis, music processing, and biomedical signal processing.

Continuous Wavelet Transform (CWT)

Continuous wavelet transforms (CWTs) offer unstable window sizes that are tuned based on frequency at the expense of time resolution; STFTs provide an excellent representation of the spatio-temporal characteristics of a signal, but are not always ideal in certain scenarios due to their fixed resolution in the frequency domain CWT provides an excellent representation of the time-domain characteristics of a signal. CWT, on the other hand, is an operation that acts linearly on the time domain signal y(t) and is given by Eq. (11)

where φ($ \frac{t-b}{a} $) (�-��)φ($ \frac{t-b}{a} $) is an extended version of the base wavelet function �(�)φ(t) with scaling and shifting applied. >0a>0 is a scaling variable that adjusts the spread of the function, and b is the time at which the signal must be analyzed, i.e., the time shift parameter.

The CWT visual representation of a signal is called a scalogram; CWT allows a more flexible view of the temporal and frequency characteristics of a signal by analyzing the signal at different scales. This makes it easier to identify local characteristics and anomalies in the signal and is especially useful for analyzing non-stationary or complex signals. Scalograms visually capture these characteristics and help to understand signals in detail, and CWT is widely used in areas as diverse as speech signal processing, seismic data analysis, and electrocardiogram (ECG) analysis.

Hilbert Huang Conversion (HHT)

Hilbert-Huang transform (HHT) is a technique for analyzing non-stationary and nonlinear signals. While many techniques can fail in the analysis of non-stationary and nonlinear systems, HHT reduces the challenges of spatio-temporal energy representation of data HHT includes two main phases: empirical mode decomposition (EMD) and Hilbert transform (HT).

The conversion process involves several steps. First, EMD is used to obtain the intrinsic mode function (IMF) from the signal. Next, a Hilbert transform is applied to each IMF component. Finally, the instantaneous frequency and amplitude are computed.

HHT allows a more detailed analysis of the temporal and frequency characteristics of a signal by decomposing the signal into IMFs and applying a Hilbert transform to each. HHT is a particularly useful tool for the analysis of complex and time-varying signals.

Other Conversion Methods

In addition to the aforementioned methods for converting IoT time series data, alternative techniques commonly used to address different types of problems are also provided in the literature. These methods play an important role in the transformation of IoT time series data. Notable approaches include a combination of data normalization and matrix transformation, direct drawing methods, Gaussian Mixture Regression (GMR), grayscale encoding (GS), grayscale image representation, RGB color image transformation, and wavelet distributed image (WVI) methods. These techniques have gained popularity in the literature because of their effectiveness in transforming and improving the analysis of IoT time series data.

• Garcia et al. modified GS by choosing the lower and upper bounds of the original formula according to GAF encoding rather than minimum and maximum scaling.
• Wang et al. used the direct drawing method, which converts the signal directly into a 2D spectral map using the plt function in Python's Matplotlib package. The direct drawing method has higher accuracy than GAF and MTF after STFT.
• The main idea of GS is to convert a time-series raw signal into an image. The time-series raw signal completes the pixels of the image in sequence.
• Wen et al. reconstructed GS using CNN for fault diagnosis in manufacturing systems.
• A transformation method combining data normalization and matrix transformation was used for the 2D image representation: 1D time series data is first normalized by [0,1] with n features, and the features are arranged in mxm matrix format. Finally, this matrix is resized to 28x28 pixels and stored as a grayscale or RGB color image.
• Voltage-current (VI) trajectories can be converted to pixelated VI images (nxn matrix) by meshing the VI trajectories.
• Qu et al. generated 2D load signatures according to corresponding features of the signal based on Weighted Voltage-Current (WVI) trajectory images.

Table II. Summary of image conversion applications by data type

Image Transformation in IoT Applications

The IoT includes a variety of areas where time-series data is frequently used. Time series data are data types that contain a series of data points collected at regular intervals over time. Table II summarizes existing research categorized into nine IoT domains. Some of the IoT domains where time-series data is frequently used are listed below:

Table III. research summary of image conversion techniques in the IoT application domain.

Security and Privacy

The IoT security and privacy field focuses on addressing the challenges and risks associated with ensuring the confidentiality, integrity, availability, and privacy of IoT systems, devices, and data Security in the IoT field includes the implementation of measures to prevent unauthorized access, data leakage, and Privacy in the IoT sector refers to the protection of an individual's personal information and their control over how it is collected, used, and shared by IoT systems. control over how it is collected, used, and shared by IoT systems.

IoT time series data plays an important role in the security and privacy arena by providing valuable insights into behaviors, patterns, and anomalies within IoT systems. In the context of IoT security and privacy, time series data can be used for a variety of purposes, including intrusion detection, unauthorized access detection, anomaly detection, and security analysis prediction. The data can be used to

• Baldini et al. proposed a method for authenticating IoT wireless devices based on radio frequency (RF) emissions. The proposed approach combines CNN and RP (RP-CNN) and was tested on an RF emission data set, which is experimental data collected from 11 IoT devices.
• Lyu et al. proposed an intrusion pattern recognition framework, based on GAF and CNN, which achieved a fast response time of 0.58 seconds and a high recognition accuracy of 97.57% for six types of fiber optic intrusion events.
• Zhu et al. developed a monitoring system to detect anomalous traffic and vulnerability attacks in IoT applications. In this system, time-series data was converted to GAF graphs and a combined CNN and LSTM model was used for traffic monitoring.
• Bertalanic et al. proposed a novel resource-aware approach based on image transformation and deep learning for anomaly detection at the wireless link layer. Time series data were transformed into images using RP and GAF, and experiments showed that RP outperformed the GAF method by up to 14%.

Energy Management

IoT enables energy consumption monitoring and control, smart grid management, and integration of renewable energy sources, helping to optimize energy distribution, reduce waste, and improve sustainability. Below are some examples of energy-related research using IoT technologies.

• Fahim et al: proposed a "Time-series to Image (TSI)" model for detecting abnormal energy consumption in residential buildings. This study focuses on analyzing univariate time-series energy data for very short-term analysis. The proposed model utilized the One-Class Support Vector Machine (OCSVM) as a classifier and the MTF as a converter to convert univariate time-series data into images.
  
• Estebsari and Rajabi: proposed a hybrid model based on CNN and image encoding methods for single housing loads. three different image encoding methods, RP, GAF, and MTF, were applied to historical load time series data and RP performed the RP showed the best performance among the three encoding methods.
  
• Alsalemi et al.: developed a new GAF classifier based on EfficientNet-B0 for the classification of edge Internet energy applications. The lightweight 2D energy consumption classifier was designed to run on the ODROID-XU4 platform.
  
• Copiaco et al: proposed a 2D pre-trained CNN model for detecting anomalies in building energy consumption. The model used 2D versions of energy time series signals as input to pre-trained models such as AlexNet and GoogleNet as features for a linear support vector machine (SVM) classifier.
  
• Chen and Wang: We proposed an edge computing architecture for load-aware tasks in the field of non-intrusive load monitoring (NILM) to reduce data transmission volume and network bandwidth requirements. We also developed a GAF-based color encoding method to construct load signatures for consumer electronics.
  
• Qu et al.: constructed three 2D load signatures based on WVI, MTF, and I-GAF and designed a new Residual Convolutional Neural Network for appliance identification in NILM The EN-SE-RECNN performed better than other models The EN-SE-RECNN showed better performance than other models.

These studies provide useful insights in energy consumption monitoring, anomaly detection, and load recognition by converting energy time series data into images and utilizing deep learning and other advanced analytical methods.

Health care

In healthcare applications utilizing IoT technologies, time-series data can help monitor patient vital signs, analyze health trends, predict disease outbreaks, and optimize healthcare resources. Below are some examples of healthcare-related research using IoT technology.

• Zhou and Kan: A tensor-based framework for ECG abnormality detection for Internet of Health Things (IoHT)-based cardiac monitoring and smart management of cardiac health was developed. Multichannel ECG signals were converted to 2D images using GADF.
• Sreenivas et al.: proposed a CNN model for arrhythmia classification in dual-channel ECG signals; GAF and MTF were used to convert ECG time series signals into images, showing that the GAF model achieved higher accuracy than MTF.
• Anjana et al.: proposed a CNN model based on an image encoding approach to classify human emotions using EEG signals. Spectrogram, scalogram, and HHT were used to convert EEG signal data into images, and it was shown that image encoding of the scalogram provided the best classification accuracy.
• Paula et al.: proposed a 2D kernel-based CNN architecture for classification of stable-state visual evoked potential (SSVEP) signals; EEG data were encoded into images using GADF, GASF, MTF, and RP, with GADF and RP showing consistently high performance.
• John et al: developed a wireless sensing-based cardiac monitoring system for accurate diagnosis of cardiac disease. The system used MQTT for long distance transmission and HTT for data preprocessing and feature extraction.
• Sharma et al: introduced an ontology-based patient monitoring system for early remote detection of COVID-19. The proposed system relied on an alarm-enabled bio-wearable sensor system that uses 1D biomedical signals such as ECG, PPG, temperature, and accelerometers.
• Chen et al.: proposed an edge computing architecture for load recognition tasks in the field of non-invasive load monitoring (NILM).

These studies combine image transformation techniques with deep learning to analyze ECG, EEG, and other medically relevant time series data to provide useful insights in areas such as disease diagnosis, anomaly detection, and emotion recognition.

For Industry

Industrial IoT (IIoT) involves connecting industrial equipment, machines, and systems to enable data monitoring, analysis, and optimization in manufacturing, energy transportation, and other industrial sectors. In industrial environments, time-series data can help monitor equipment performance, predict failures, optimize maintenance schedules, and improve overall operational efficiency. in IIoT, various image encoding methods are commonly used to provide efficient and intelligent fault diagnosis In the IIoT, different image encoding methods are commonly used to provide efficient and intelligent fault diagnosis.

• Wang et al.: proposed a framework for fault diagnosis of single-channel and multi-channel bearing signals. Combined spectral map information fusion and CNN for fast fault diagnosis.
• Zhang et al.: proposed a novel fault diagnosis method combining GAF, extreme learning machines (ELMs), and CNNs. Different encoding methods were explored and the effectiveness of selected encoding techniques for pattern recognition was demonstrated.
• Santo et al.: developed a model combining time series encoding techniques and CNNs for predictive maintenance; evaluated four major encoding techniques, including RP, GAF, MTF, and wavelet transform.
• Ferraro et al.: developed an efficient predictive maintenance method that uses GAF to convert time-series data into images and leverages deep learning strategies to predict the health of hard disk drives (HDDs).
• Jiang et al.: proposed an MTF-CLSTM method combining MTF, CNN, and LSTM to predict product quality in wire electrical discharge machining (WEDM); MTF was used to convert dynamic WEDM manufacturing conditions into images.
• Garcia et al.: explored six encoding methods (GAF, MTF, RP, GS, spectrogram, and scalogram) and their modifications to increase robustness to variations in large data sets when converting time series signals to images.
• Bai et al.: failure using GAF and MTF to convert 1D electrical parameters into 2D images and Deep Convolutional Generative Adversarial Network (DCGAN) to process insufficient data samples of electrical parameters from oil wells A diagnostic method, Time-series Conversion-DCGAN (TSC-DCGAN), was proposed.

These studies leverage image encoding techniques such as GAF, MTF, and STFT to provide useful insights in areas such as failure diagnosis, quality prediction, and maintenance management in IIoT environments. These techniques make complex industrial time series data more manageable and enable more efficient analysis and decision making.

Environmental Monitoring

IoT devices are used to monitor and manage environmental conditions such as air quality, water quality, pollution levels, and conservation of natural resources. These solutions help protect the environment and sustainable practices.

• Abidi et al: proposed a framework for classification of land use/land cover (LULC) mapping based on 2D encoded multivariate satellite image time series (SITS). In this study, multivariate SITS data were transformed into 2D images by GADF, GASF, MTF, and RP. The results showed that the RP technique performed better than all encoding techniques. Additionally, the combination of 2D encoding techniques achieved better performance than the use of stand-alone encoding methods.

This research provides a new approach to land use and land cover mapping by utilizing IoT technology to collect and analyze environmental data. Image encoding technologies will enhance environmental monitoring and sustainability efforts by extracting richer information from time-series data and enabling more accurate classification and analysis. Such approaches have the potential to play an important role in contributing to environmental protection and sustainable development goals.

Smart building

Smart buildings increase occupant comfort, reduce energy consumption, improve safety and security, and optimize building operations and maintenance. Time-series data in smart buildings is used to monitor and control various building systems such as HVAC (heating, ventilation, and air conditioning), lighting, and occupancy.

• Sayed et al.: presented an approach to occupancy detection using environmental sensor data such as temperature, humidity, and light sensors. In this study, multivariate time series data were transformed into grayscale and RGB images using image transformation methods to better encode and capture relevant features. This method covered data normalization and matrix transformation, unlike commonly used methods such as GAF. Results showed that grayscale images provided the right balance of accuracy and training time compared to color images.

This study demonstrates the use of sensor data in smart building environments and shows how image transformation techniques can be used to enhance the analysis and feature extraction of time series data, especially in specific applications such as occupancy detection. This approach can contribute to more efficient monitoring and control of building systems and play an important role in achieving objectives such as reducing energy consumption and improving occupant comfort.

Transportation & Logistics

IoT applications have the potential to transform the transportation and logistics sector, including fleet management, vehicle tracking, route optimization, cargo monitoring, and driver safety. This will enable intelligent decision making, reduce costs and improve customer experience.

• Huang et al: developed a new method for reconstructing missing values in traffic sensor data, Traffic Sensor Data Imputation GAN (TSDIGAN). In this study, time-series traffic data were processed using Gramian Angular Summation Field (GASF) and converted to image format using CNN to complement missing values.

These developments have greatly advanced the application of IoT in the transportation and logistics industry. Fleet management can analyze time-series data from vehicles to predict maintenance needs, optimize fuel consumption, and improve route planning. Vehicle tracking and route optimization reduce travel time and fuel consumption, lowering operating costs and emissions. Cargo monitoring ensures the safety and integrity of goods in transit, and driver safety applications can detect risk behaviors and provide warnings to prevent accidents.

Thus, the use of IoT in transportation and logistics offers significant benefits in terms of operational efficiency, safety, and customer satisfaction, and advances in advanced data analytics and IoT technologies such as those by Huang et al. are paving the way for a more connected, efficient, and sustainable transportation industry.

Wearable terminal

Wearable devices focus on integrating technology into portable devices that can be worn by individuals. These devices are equipped with sensors, connectivity, and computing power to collect data, interact with their environment, and provide personalized experiences. Wearable devices incorporate a variety of sensors to collect data about the user and their environment, including accelerometers, heart rate monitors, GPS, and temperature sensors. They are also connected to other devices and networks through wireless technologies such as Bluetooth and Wi-Fi.

Wearable devices provide individuals with convenient access to personalized data and experiences to monitor their health, improve fitness, and connect in a more seamless and discreet manner. as IoT and wearable devices advance, sensor-based human behavior awareness (HAR ) is becoming increasingly important for its convenience and privacy properties.

• Xu et al.: presented two improvements based on GAF and deep CNN for HAR. The results of this study showed that the developed model was able to efficiently extract multi-scale features and improve the accuracy of action recognition by exploiting the properties of the GAF algorithm and combining it with the structure and advantages of CNN, residual learning, and extended convolution.

Thus, advances in wearable devices and IoT technology have made human behavior recognition using sensor data an important area of focus, providing new ways to use wearable devices to manage health, improve fitness, and enhance quality of life.

Other

Various studies in the IoT field have used data obtained from different IoT disciplines. These studies have tested the impact of the proposed method on data sets obtained from diverse disciplines.

• Yang et al.: used the well-known MTS data sets Wafer and ECG for classification of 1D signals; MTS data were converted to 2D images by applying GASF, GADF, and MTF, and these images were concatenated as RGB input channels for the ConvNet classification model. It was concluded that the choice of encoding method did not significantly affect the prediction results.
• Jiang et al.: evaluated the Adaptive Dila-DenseNet (ADDN) model for classification of UTS and MTS data across 24 benchmark IoT data sets.UTS and MTS data were converted to GM images utilizing GAF and MTF methods and fed into the ADDN model. supplied to the ADDN model.
• Quan et al.: investigated the impact of different feature construction and fusion methods on time series classification results. In this study, three images encoded in GAF, MTF, and RP were superimposed as GMR images as three-channel data input. In addition, 1D multiscale features and 2D image features were fused using two different methods, including Squeeze-and-Escitation (SE) and Self-Attention (SA) feature fusion architectures.
• Hasan et al.: introduced a digital twin-based approach to sensor fault detection. A Generative Adversarial Network (GAN) method was used to create a digital representation of the sensor, and the GAN was trained on images obtained from a time series using GAF.

These studies utilize image encoding techniques such as GAF, MTF, and RP to convert time series data into images and apply them to tasks such as classification, prediction, and fault detection in deep learning models. This enables effective data analysis and decision making in various areas of the IoT.

Research Issues and Future Directions

Converting time-series data to images is gaining attention to facilitate IoT data analysis, but there are several challenges associated with this conversion process.These major challenges and potential solutions are presented below for researchers.

1. Noise and missing values: IoT time series data often suffer from noise and missing values due to sensor failure or network problems. This can adversely affect image quality. Missing value completion methods such as image inpainting models and GAN-based models can be used to handle missing gaps.
   Large Data Encoding: Encoding large IoT time series data can be computationally expensive and memory intensive. Optimal image dimensions need to be determined to capture meaningful patterns without overwhelming computational resources.
   Information loss: The process of converting time-series data to images can result in information loss due to the compression of temporal information into a 2D representation. Balancing the trade-off between dimensionality reduction and information loss is a key challenge in this area.
   MTS Data Handling: IoT time-series data may involve multiple variables and sensors, forming MTS. Effective fusion techniques need to be developed to convert MTS data into the appropriate format.
   Real-time or quasi-real-time image representation: Real-time or quasi-real-time image representation in a dynamic IoT environment is challenging. If the image conversion process takes longer than the interval time, the system may eventually fail. Edge devices provide remarkable computational resources for faster real-time decision making.

To address these challenges, there is a need to develop more robust transformation technologies, combine edge and cloud architectures, implement effective fusion technologies, and increase hardware resources. This will enable more efficient and effective analysis and decision making of time-series data in IoT applications.

Conclusion

Methods for converting time-series data into images have become widely popular in recent years, but adoption of these techniques in the IoT sector is still in its infancy and is expected to become commonplace in most IoT sectors in the near future. This study presents a comprehensive review of image conversion techniques used in various IoT areas, including smart buildings, industrial environments, energy management, healthcare, and security. Existing studies are categorized based on their encoding techniques, IoT application areas, and data types.

In the literature, various transformation techniques have been applied to univariate and multivariate time series IoT data. These transformation techniques are usually combined with fusion techniques for multivariate time series IoT data. Among the techniques used, GAF and MTF are the most commonly used image transformation techniques, especially in areas such as energy management, healthcare, and industrial applications for anomaly detection, fault diagnosis, and time series classification.

In addition, the paper discusses related challenges, open problems, and future research directions. As the adoption of these technologies continues, it is hoped that new possibilities will open up for the analysis and processing of IoT data, enabling more effective and efficient applications.

• 
• 
• 

Categories related to this article

• Article
• Image Recognition And Analysis
• Time Series
• Time-series
• Internet Of Things