Explainability Techniques To Enhance Predictive Models Of Manufacturing Quality

Explainable.AI 29/08/2024

3 main points

✔️ Using explainability techniques to improve the accuracy of manufacturing quality prediction models and reduce manufacturing costs
✔️ Selecting important features and removing unnecessary features to improve efficiency and comprehension
✔️ Proposing a highly accurate prediction method using explainability techniques under limited data volume conditions

Enhancing Manufacturing Quality Prediction Models through the Integration of Explainability Methods
written by Dennis Gross, Helge Spieker, Arnaud Gotlieb, Ricardo Knoblauch
[Submitted on 27 March 2024]
Comments: Accepted by arXiv
Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Computers and Society (cs.CY); Machine Learning (cs.LG)
code：

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

This study proposes a method for using explainability techniques to improve the performance of machine learning (ML) models. This method has been used in milling quality prediction, a process that first trains ML models and then uses explainability techniques to identify unwanted features and remove them to improve accuracy. This approach is expected to reduce manufacturing costs and improve the understanding of ML models. This study demonstrates that explainability techniques can be useful in optimizing and explaining predictive models in the manufacturing sector.

Introduction

Milling is a machining process that removes material to produce a desired shape or surface finish. In this process, a cutting tool called a milling cutter rotates at high speed and removes material while moving the workpiece. The workpiece is fixed to a table that can be moved in multiple axes, allowing machining in various directions and angles (Fertig et al., 2022). Energy consumption in milling can vary widely depending on the setting and material, but it is generally considered an energy-intensive process. If quality problems can be predicted and prevented, energy consumption can be reduced and waste from producing defective parts can be reduced (Pawar et al., 2021).

Machine learning (ML) models can identify patterns and structures in data and make predictions without being directly directed by a program. These models are useful tools for predicting the final quality of milling operations and can improve the efficiency and reliability of the manufacturing process (Mundada and Narala, 2018). However, experimental data from milling operations are very expensive, so available data is often scarce, making it difficult to train ML models to predict workpiece quality (Postel et al., 2020).

Furthermore, when complex ML models are used, especially deep neural network models, their inner workings are opaque and their "black box" nature can be problematic. This problem arises because the predictions of these models are difficult to understand and therefore cannot be fully understood by practitioners and stakeholders in milling quality prediction (Kwon et al., 2023).

This study proposes a method to improve the performance of ML models by using explainability methods to elucidate the prediction mechanism of ML models and to optimize them. By using explainability methods, optimization can be effectively advanced by identifying important features in ML model predictions and eliminating unnecessary features (Bento et al., 2021; Sun et al., 2022).

Related Research

The application of machine learning (ML) to manufacturing and machining tasks has been the focus for at least a decade (Kummar, 2017). For example, ML was initially used to optimize turning operations (Mokhtari Homami et al., 2014), predict stable conditions for milling operations (Postel et al., 2020), estimate bore quality (Schorr et al., 2020), classify defects using ML-driven surface quality control ( Chouhad et al., 2021), and others.

However, it is only recently that the application of explainable AI (XAI) methods to manufacturing processes has begun to attract attention (Yoo and Kang, 2021; Senoner et al., 2022). The ongoing European XMANAI project (Lampathaki et al., 2021) aims to assess the capabilities of XAI in various sectors of manufacturing. In particular, it has shown the potential for successful application of XAI in the field of fault diagnosis (Brusa et al., 2023).

There also exist studies that focus on feature selection of datasets without directly considering ML models (Bins and Draper, 2001; Oreski et al., 2017; Venkatesh and Anuradha, 2019). This study explores the possibility of using XAI to improve quality prediction models by eliminating unnecessary sensors; while approaches to improve ML model performance through explainability methods are known in the context of explainable ML (Bento et al., 2021; Sun et al., 2022; Nguyen and Sakama, 2021; Sofianidis et al., 2021), this is the first time XAI has been used to model milling quality prediction. In particular, the approach of using XAI methods to improve quality prediction models of milling processes by identifying and eliminating unwanted features is novel.

methodology

This study proposes a way to improve the performance of machine learning (ML) models by using the following steps to improve the performance of machine learning (ML) models using explainability techniques.

1. training of ML models

Initially, the ML model is trained using the given data set. Three models are used in this study: decision tree regression, gradient boosting regression, and random forest regression. These models require less data and are easier to interpret than neural networks.

2. application of explainability method

Apply explainability methods to the trained ML model and dataset to identify features that are important to prediction accuracy. In this step, we rank the importance of the features and train new models with increasing number of important features.

3. feature selection

Based on the importance of the features, train a new model using only the most important features. This process improves the performance of the ML model.

Machine Learning Model

decision tree regression model

The decision tree regression model divides the input space into different regionsand fits a simple model (usually constant) to the training samples within each region. Theprediction $𝑦^ for$ a new input $𝑥x$ is expressed as

where $𝑐𝑚cm$ is a constant fitted to the region $𝑅𝑚R$ m, $𝑀M$ is the number of regions and $𝐼\cdotI\cdot$ is the indicator function.

Gradient boosting regression model

The gradient boosting regression modelcombines several weak models to optimize theloss function. Starting from aninitial approximation_F0(x),the modelis updated by adding the weak model_hmx(x) as follows

where $𝛼α$ is the learning rate and _hm (x) is the weak learner to correct for errors in the previous model.

Random Forest Regression Model

The random forest regression model makes its final prediction by training multiple decision trees and averaging their predictions. The final prediction for a new input $𝑥x$ is expressed as

where $𝑇T$ is the total number of trees and _yt (x is the t-th tree prediction.

$𝑦^(𝑥)=1𝑇∑𝑡=1𝑇𝑦𝑡(𝑥)$

accountability measure

Permutation importance of features

Feature permutation importance is an accountability method for evaluating the important features of a model. It measures the importance of each feature by randomly replacing certain features and monitoring changes in model performance.

Shapley Value

The Shapley value is derived from cooperative game theory and assigns a fair value based on the marginal contribution of each participant. In the context of machine learning models, it quantifies the contribution of each feature to the prediction.

Case study

In this study, the proposed method was applied using the dataset generated by ENSAM. The details are described below.

Objective

The objective of this case study is to develop a predictive model for each quality indicator. This includes not only training the models, but also clarifying the reasons for the predicted results, as well as identifying and removing unnecessary features. This step aims to minimize installation and maintenance costs by reducing redundant sensors, thereby optimizing resources and reducing costs.

Figure 1: Milling machine processing a workpiece

Data preprocessing

Because this study deals with data with variable time series lengths, boxplot values were computed for each time series. In addition, the metadata in the data set includes various experimental parameters.

Training Machine Learning Models

In this study, we trained a decision tree regression, a gradient boosting regression, and a random forest model to predict each quality indicator. Figure 2 shows the overall model inputs and outputs. A five-part cross-validation was employed to train each model. This method divides the data into five equal parts, with four parts (80%) used for training and the remaining one (20%) for testing at each iteration. This process is repeated five times, with each of the five parts serving as a test set once. The performance of the model is averaged over the five iterations to provide a more robust evaluation.

Figure 2: The ML prediction model receives boxplots (time and frequency domain) and machine configuration parameters and outputs quality measures.

Analysis

We analyze the proposed method. First, we evaluate the performance of the ML model, then we analyze the prediction mechanism of the ML model, and finally we evaluate the impact of removing features on the performance of the ML model.

Assessing the model's predictive quality

The purpose of this study is to evaluate the forecasting accuracy of three ML models: gradient boosting regression, decision trees, and random forests. The primary metric used to assess the quality of the forecasts is the mean absolute error rate (MAPE). A forecast is considered high quality if its MAPE is less than 5%.

Setup: using a preprocessed data set of 100 samples.
Run: Using the preprocessed dataset, we trained three different machine learning models: gradient boosting regression, decision trees, and random forests, and measured MAPE using k-partition cross-validation.
FINDINGS: Using these ML techniques, we were able to predict Rdq on a complete set of quality features with an error rate of less than 5%. Specifically, we achieved error rates of 4.58% for the gradient boosting regression model and 4.88% for the random forest model.

Understanding the predictive mechanisms of ML models

In this study, we evaluated the importance of each attribute in predicting quality indicators.

Setup: We focused on the gradient boosting regression model that showed the best performance for the trained models.
Run: Applied the permutation importance and Shapley values for the features.
Finding: we observed that different explanatory methods indicate different reasons. For example, permutation feature importance highlighted fa_ts_max as the more important feature compared to the Shapley value (see Figure 3).

Figure 3: Visualization showing feature importance ranks for Rdqmaxmean predictions by FPI (feature permutation importance permutation) and SHAP (Shapley value) methods.

Performance Improvement

This experiment explored the possibility of integrating explainability methods into the ML model development process to improve model performance.

Setup: variables in the ML model were classified in descending order based on feature importance, and a new model was trained on each trial, varying the proportion of top features (p).
Run: We attempted to improve the performance of the model using only the important features.
Finding: Integrating only the most important features into the training dataset improved the performance of the ML model. For example, by selecting only the top 20% most important features in terms of permutation importance, we improved the MAPE from about 4.58 to 4.4.

Figure 4: Use of different proportions of the most important features based on different methods in Rdq prediction. FS is feature selection.

Discussion

This case study demonstrates the benefits of an explainable machine learning (ML) approach to manufacturing quality prediction models. Explainability scores (like feature importance) are used to interpret the relevance of each feature to the predictive power of the model. This interpretation can be used by human experts to analyze the trained model and verify whether high importance features are meaningful for the prediction task.

ML models can reveal new relationships between input features and predicted targets, but in a quality prediction setting, overreliance on certain features can be an indicator of learning false correlations due to lack of data. Explainability methods can serve as tools for model validation and human inspection.

Furthermore, it has been shown that the model can be improved by removing low-ranked features. Specifically, the accuracy of the model was improved by keeping only the most important features, as shown in Figure 4. This approach not only improves the accuracy of the prediction, but also reduces the number of sensors required for the prediction, thus reducing the computational cost of the prediction. When performing real-time quality forecasting during the manufacturing process to detect potential defects and deviations from plan, it is important to minimize the time required for forecasting and to increase the frequency of forecasts.

Also, when designing a prototype manufacturing machine, it is important to evaluate the predictive model in the early stages to determine the final set of sensors for the machine. Although the prototype machine is equipped with many sensors, only relevant sensors are selected after evaluation of the predictive model.

Furthermore, the use of simple and interpretable models (Breiman, 2001; Rudin et al., 2022) may be beneficial for the development of quality prediction models in the manufacturing industry. However, there is a tradeoff between simplicity and accuracy (Occam's dilemma): the simpler the model, the lower the accuracy. In the case study, this tradeoff was also seen in the error difference between simple decision trees and more complex gradient boosting trees and random forests. Reducing the number of features using explainability techniques again reduced the complexity of the model and made the final model more interpretable.

Conclusion

This study demonstrates the potential of combining machine learning (ML) and explainability techniques to improve the performance of surface quality prediction models in the manufacturing industry. Despite the limited availability of data, feature selection based on explainability techniques was able to increase the effectiveness of ML models with small amounts of data.

Future work will extend the explainability method to manufacturing processes other than milling to create a more comprehensive prediction system. In addition, these ML models will be utilized as a digital twin of the physical machine and applied to parameter optimization methods for new developments. This integration will not only improve the accuracy of the models, but also allow for real-time fine-tuning of machine operations, which is expected to increase efficiency and reduce costs.

Categories related to this article

友安昌幸 (Masayuki Tomoyasu): JDLA G certificate 2020#2, E certificate2021#1 Japan Society of Data Scientists, DS Certificate Japan Society for Innovation Fusion, DX Certification Expert Amiko Consulting LLC, CEO