[No Need To Stain?] Style Conversion Of HE-stained Images To Other Stains In The Image Generation Network

3 main points
✔️ Converts HE stained image to another stained image by deep learning
✔️ Convert to Masson trichrome staining, PAS staining, and Jones plating silver staining to improve diagnostic accuracy
✔️ This method can be adapted to various staining methods

Deep learning-based transformation of H&E stained tissues into special stains
written by Kevin de Haan, Yijie Zhang, Jonathan E. Zuckerman, Tairan Liu, Anthony E. Sisk, Miguel F. P. Diaz, Kuang-Yu Jen, Alexander Nobori, Sofia Liou, Sarah Zhang, Rana Riahi, Yair Rivenson, W. Dean Wallace, Aydogan Ozcan
(Submitted on 12 Aug 2021)
Comments: Nature Communications.
Subjects: Computer Vision and Pattern Recognition (cs.CV)

code：  

The images used in this article are from the paper, the introductory slides, or were created based on them.

Abstract

In the pathological examination, HE staining is known as the most basic staining method. In addition to this, special staining methods can be used to obtain tissue- and disease-specific histological images. In this paper, we introduce a machine learning model that converts HE staining to special staining (Masson trichrome staining, PAS staining, and Jones plating silver staining) using tissue sections from kidney needle biopsies.

The model was built by supervised learning with the evaluation of three renal pathologists and the diagnosis of a fourth pathologist. The results showed that the virtual stained images of 58 cases improved the diagnostic accuracy of several non-neoplastic renal diseases (p=0.0095). In addition, the virtual stained images were found to be statistically equivalent to the actual histochemical stained images. This staining conversion can improve diagnostic accuracy and significantly reduce staining costs.

first of all

Histopathological evaluation is performed by looking through a microscope at a tissue section or by viewing a scanned image of a whole slide image (WSI) on a computer screen. Visual observation is the gold standard of histopathology and an important workflow in pathology, regardless of the type of disease.

Sections of tissue are stained to add color contrast, with hematoxylin and eosin (HE) staining being the most common staining method; HE staining is relatively simple and is performed in almost all cases, accounting for about 80% of human tissue staining.

In addition to HE staining, various other staining methods are used to highlight histological characteristics. For example, Masson's trichrome (MT) is used to view connective tissue, and periodic acid-Schiff (PAS) is used to examine the basement membrane. Jones methenamine sliver (JMS) provides a clearer visualization of glomerular structures. These stains allow the pathologist to recognize subtle basement membrane abnormalities and thus recognize the non-neoplastic renal disease.

Traditional histopathological workflows require not only time and money, but also laboratory infrastructure. Furthermore, when multiple types of staining methods are used, multiple tissue fragments themselves are required, and different staining procedures are required, resulting in high costs. In general, HE staining is performed first, and then special staining is performed at the discretion of the pathologist. Therefore, it takes time to perform special staining.

To address this issue, alternative contrast mechanisms such as nonlinear microscopy and UV tissue surface excitation are known. More recently, virtual staining by deep learning has been developed. The staining is based on a variety of modalities such as autofluorescence, hyperspectral, and quantitative phase. These techniques are performed on label-free, i.e. unstained sections.

In contrast, a method of converting already stained material to another stain (stain conversion) has been considered. Various staining conversions have been introduced in the literature, such as HE to MT, Ki67-CD8 to FAP-CK (fibroblast activation protein-cytokeratin) in situ hybridization. A variety of staining conversions have been described in the literature, including HE to MT, Ki67-CD8 to FAP-CK (fibroblast activation protein-cytokeratin) for in situ hybridization.

However, many of these staining transformation techniques rely on unsupervised approaches, adversarial generative networks known as CycleGANs. Such networks, which use only distribution matching loss, are prone to hallucination when applied to medical images. Generative images leading to misdiagnosis) is known to be prone to occur.

Therefore, in this paper, we introduce a supervised learning staining transformation work-frame (Figure 1). The authors validate this model by evaluating tissues from non-neoplastic renal diseases. In many clinical settings, the pathologist first makes a preliminary diagnosis with HE staining. Although treatment can be initiated with this diagnosis, a definitive diagnosis is often made with special stained images provided the next day. Therefore, the time required to provide special stained images can be shortened with this model, which is expected to lead to significant clinical improvement in urgent diseases such as crescentic glomerulonephritis and GVHD.

Figure 1: Deep learning is used to transform HE-stained images into specially stained images.

result

The validity of the stain conversion in this model was confirmed by three pathologists. There are 58 cases, and the pathologists make a diagnosis from HE sections and stain-transformed histology. They then make the diagnosis on the actual specially stained sections and compare the two diagnoses.

In the past, pathologists made a preliminary diagnosis by looking at the histological image in HE, and then applied special staining if necessary. In this study, we succeeded in skipping the special staining process by generating a special stained image, which improved the diagnostic accuracy of various non-neoplastic renal diseases.

Design and training of statin transformation networks

One CNN performs the staining transformation, but a GAN is used to generalize the input image (Style transfer network in Figure 2b).

Figure 2b. This CycleGAN transforms the image into a slightly different view of the same HE-stained image (generalization). This reproduces the coloration that varies among researchers, labs, temperatures, reagents, etc.

Figure 2a shows the network for virtual staining, with DAPI- and Texas Red-stained images as input and virtual HE-stained images as output. The actual dyed HE image is the teacher data. This network is the basis of the dye conversion network.

Figure 2c. and the output image of the Style transformation network is input to the staining transformation network, and the network is designed to finally output the special stained image.

Evaluation of stain transformation networks for kidney disease diagnoses

The evaluation of staining transformation was done as follows. First, three pathologists make the first diagnosis on HE sections only (58 cases).

Then, after a forgetting period of at least 3 weeks (to forget the first diagnosis), a second diagnosis is made on HE sections and stain-transformed special-stained sections.

After another three weeks, a third diagnosis is made, this time on HE sections and actual specially stained sections.

In summary, we provide the pathologist with HE only for the first time, HE and stain transformation images for the second time, and HE and actual special stained sections for the third time.

Finally, one pathologist, different from the three who diagnosed the disease, compiles the results of a total of three diagnoses and makes a judgment of improvement, agreement, or disagreement. Note that the correct labels used in this study were determined based on information obtained from observation methods other than those described above, such as electron microscopy images and immunofluorescence staining.

The results showed that the use of staining transformation improved the results in an average of 13 cases (22.4%). The rest of the results showed 38.3 cases (66.1%) of agreement and 6.7 cases (11.5%) of disagreement.

Figure 4a. above is a graph tabulating how the diagnosis changed between the first (HE only) and second (HE and stain conversion) times. As shown in the middle panel, they are basically in agreement. The horizontal axis is the case number; for example, the ninth case shows an improved diagnosis for all pathologists.

Figure 4b. above is a comparison of the first (HE only) and third (HE and actual special stain) diagnoses. The results are generally similar to Figure 4a, but the discordance has decreased. The results here are 25.8% improvement, 66.6% agreement, and 7.4% discordance. As expected, the diagnostic accuracy is improved when special staining is applied.

From Figures 4a and 4b, we found that special staining, whether virtual or actual, improved diagnostic accuracy over HE staining. We now tested whether there was a significant difference between the diagnostic accuracy with staining conversion and actual staining for each pathologist. The results of the chi-square test showed that the actual special staining was significantly better for two pathologists (no significant difference for the remaining one).

Figure 5 shows a case in which the diagnostic accuracy was improved by staining conversion, because HE staining does not stain the basement membrane clearly, and it is thought that staining conversion made it easier to see inflammatory cells in the tubules (arrows).

This is another case where there was an improvement in diagnosis. The bottom row shows the actual special staining image (Note: For actual special staining, the section used for HE cannot be used. (Note: For actual special staining, the section used for HE cannot be used; therefore, the morphology of the section is different from that of the HE image because the section before and after is used for special staining. (Conversely, all three stain-transformed images are identical.)

The above is a case where staining conversion rather missed the diagnosis; the pink color of the PAS stain (fibrin clot) was difficult to see, which may have resulted in a discordant diagnosis.

Conclusion

In this study, we focused on the conversion from HE staining to other special stains. However, the method can be adapted to other conversions, for example, from special stains to HE, and from immunofluorescence stains to special stains.

This method eliminates the need for the pathologist to prepare sections for special staining and allows the visualization of different tissue components. It is also fast (1.5 ^mm2/s on 2 GPUs) in performing staining conversions, saving labor, time, and chemicals. Not only that, but it will also bring great benefits to the medical system.

• 
• 
• 

Categories related to this article

• Medical Care
• Article
• Image Generation
• Medical