Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 840  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 




 
Table of Contents    
RESEARCH ARTICLE
J Pathol Inform 2021,  12:29

Improving algorithm for the alignment of consecutive, whole-slide, immunohistochemical section Images


1 Department of Pathology, Fu Jen Catholic University Hospital, Fu Jen Catholic University; School of Medicine, College of Medicine, Fu Jen Catholic University; Department and Graduate Institute of Pathology, National Taiwan University Hospital and National Taiwan University College of Medicine, Taipei, Taiwan
2 Department of Computer Science and Information Engineering, National Taiwan University; Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University; MOST Joint Research Center for AI Technology and All Vista Healthcare, Taipei, Taiwan
3 Department of Pathology, Fu Jen Catholic University Hospital, Fu Jen Catholic University, New Taipei City, Taiwan
4 Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan

Date of Submission03-Dec-2020
Date of Decision29-Apr-2021
Date of Acceptance01-Jun-2021
Date of Web Publication03-Aug-2021

Correspondence Address:
Dr. Chiao-Min Chen
Department of Computer Science and Information Engineering, National Taiwan University, Taipei 10617
Taiwan
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jpi.jpi_106_20

Rights and Permissions
   Abstract 


Background: Accurate and precise alignment of histopathology tissue sections is a key step for the interpretation of the proteome topology and cell level three-dimensional (3D) reconstruction of diseased tissues. However, the realization of an automated and robust method for aligning nonglobally stained immunohistochemical (IHC) sections is still challenging. In this study, we aim to assess the feasibility of multidimensional graph-based image registration on aligning serial-section and whole-slide IHC section images. Materials and Methods: An automated, patch graph-based registration method was established and applied to align serial, whole-slide IHC sections at ×10 magnification (average 32,947 × 27,054 pixels). The alignment began with the initial alignment of high-resolution reference and translated images (object segmentation and rigid registration) and nonlinear registration of low-resolution reference and translated images, followed by the multidimensional graph-based image registration of the segmented patches, and finally, the fusion of deformed patches for inspection. The performance of the proposed method was formulated and evaluated by the Hausdorff distance between continuous image slices. Results: Sets of average 315 patches from five serial whole slide, IHC section images were tested using 21 different IHC antibodies across five different tissue types (skin, breast, stomach, prostate, and soft tissue). The proposed method was successfully automated to align most of the images. The average Hausdorff distance was 48.93 μm with a standard deviation of 14.94 μm, showing a significant improvement from the previously published patch-based nonlinear image registration method (average Hausdorff distance of 93.89 μm with 50.85 μm standard deviation). Conclusions: Our method was effective in aligning whole-slide tissue sections at the cell-level resolution. Further advancements in the screening of the proteome topology and 3D tissue reconstruction could be expected.

Keywords: Cell-level resolution, Image registration, Immunohistochemistry, Multidimensional graph-based registration, Proteome topology, Three-dimensional tissue reconstruction, Whole-slide images


How to cite this article:
Liang CW, Chang RF, Fang PW, Chen CM. Improving algorithm for the alignment of consecutive, whole-slide, immunohistochemical section Images. J Pathol Inform 2021;12:29

How to cite this URL:
Liang CW, Chang RF, Fang PW, Chen CM. Improving algorithm for the alignment of consecutive, whole-slide, immunohistochemical section Images. J Pathol Inform [serial online] 2021 [cited 2021 Sep 24];12:29. Available from: https://www.jpathinformatics.org/text.asp?2021/12/1/29/323081




   Introduction Top


Automatic and precise alignment of consecutive whole-slide histopathology section images is a key process leading to the future cell-level three-dimensional (3D) tissue reconstruction and topographical proteomics. However, only few studies focused on this topic.[1],[2],[3],[4],[5],[6] The bottlenecks were attributed to the following four factors: (1) a ×10 magnified, whole-slide histopathology image was at average 32,947 × 27,054 pixels in size, and at its maximal resolution may reach 100,000 × 100,000 pixels. Dealing with data of this size was extremely challenging using the common computing hardware;[3] (2) currently, most histopathology slides are manually made, thereby introducing various distortions to the tissue section, such as cuts, folds, tearing, and local tissue stretch or compression;[1],[4] (3) each whole-slide image was unique and slightly different than each other in spatial coordinates, and they were cut in a continuous manner with 4-μm spacing, thereby making the assessment of alignment adequacy difficult, as no ground-truth image was available in each level of the whole-slide image series; and (4) for nonglobal stained slides, such as immunohistochemical (IHC) slides, the images produced from different antibodies were usually viewed as independent objects by the current established registration methods and were, thus, difficult to be aligned, particularly in sections wherein the stained proteins are highly expressed in contrast to the low-counterstained background. Several image-registration models and algorithms were proposed in few previously conducted studies. They mainly addressed the first three above-mentioned issues and significantly improved the accuracy and precision of aligning serial neighborhood, globally-stained, histopathological section images.[1],[2],[3],[4],[5],[6] However, in daily laboratory usage, efficacy issues were still common, and the alignment and concatenation of multiple, differently and strongly stained IHC whole-slide images usually failed. In this study, we improve the current method to align consecutive, whole-slide gigapixel histopathology images, thereby enhancing the program efficiency and addressing the issue regarding the alignment of multiple, nonglobally and differently stained IHC section images. The Hausdorff distance was used to measure the effectiveness of image alignment.[1] We prove that our method can successfully achieve image alignment at a near cell-level resolution, thereby not only leading to the future cell-level 3D tissue reconstruction but also displaying the ability to incorporate the topographical protein information into the reconstruction.


   Materials and Methods Top


Whole-slide, 4 μm thick, serial IHC histopathology section images were prepared from formalin-fixed, paraffin-embedded tissues of the skin, breast, stomach, prostate, and soft tissue using 21 different IHC antibodies [Figure 1] for selected images of p63, CK5/6, EMA, CK7 and HER2, [Supplemental Table 1] for staining conditions, clones and methods for all used antibodies, and [Supplemental Figure 1] for set design], with an average image dimensions of 32,947 × 27,054 pixels. IHC staining was performed in a Leica-Bond Max automated immunostainer (Leica, Taipei, Taiwan), and whole-slide images (MIRAX Virtual Slide File., mrxs, binary, Carl Zeiss MicroImaging) were generated using a 3DHISTECH Pannoramic 250 Flash III scanner (3DHISTECH Ltd., Budapest, Hungary) at ×400 resolution using a ×20 objective, with a 24-bit red, green, and blue (RGB) format and 0.19 um/pixel resolution. The images were then viewed by CaseViewer (ver. 2.3 or above, https://www. 3dhistech.com/research/software-downloads/) and exported as. TIF files with a 1:4 conversion (equivalent to a ×100 view or ×10 objective in practice, a resolution of 0.76 μm/pixel) for further computation. This study was approved by the Institutional Review Board of the National Taiwan University Hospital (201412155RIND) and Fu Jen Catholic University (C108102).

Figure 1: Selected serial immunohistochemical histopathological section images scanned at ×10 resolution: (a) p63, (b) CK5/6, (c) EMA, (d) CK7, and (e) HER2

Click here to view


The general flowchart of the proposed method is shown in [Figure 2]. The input images contained a reference (Sequence 0) image and translated (Sequences 1–n with phase correlation, scale/rotation recovery on RGB transformation) high-resolution images (HR0 and HTn) and 1× low-resolution images of 3295 × 2705 pixels (LR0 and LTn) rescaled from HR0 and HTn. The alignment began with the initial alignment (preprocessing) of HR0 and HTn (object segmentation and rigid registration) and nonlinear registration of LR0 and LTn, followed by the multidimensional graph-based image registration of the segmented patches, and finally, the fusion of deformed patches for inspection. To assess the alignment accuracy, a graph-based method was adapted to check the distance shift from the local highest and lowest probability points converged from the neighborhood regions (Hausdorff distance).[1]
Figure 2: Flowchart of the proposed multidimensional, graph-based image-registration method. The alignment began with the initial alignment of high-resolution reference and translated images (object segmentation and rigid registration) and nonlinear registration of low-resolution reference and translated images, followed by the multidimensional graph-based image registration of the segmented patches, and finally, the fusion of deformed patches for inspection

Click here to view


Initial alignment

The initial alignment was used to reduce the object deviations between serial whole-slide IHC images. The staining intensities from different antibodies between each image were normalized by the Gamma correction.[7] Foreground object preidentification was performed to detect the extent of staining in the tissue according to the reaction intensity generated by the Otsu thresholding algorithm,[8] while the noise were removed. To correctly identify the surrounding nonstained region, a segmentation template was generated using an image matting and segmentation algorithm[9] and applied to segment the tissue based on the result of the foreground object preidentification. The segmentation template differentiated an image into three regions, namely foreground (object), background, and unknown regions. The ring region collected from the outer 50 pixels from the Otsu foreground region comprised the unknown region, and it was then verified by an image matting algorithm defined as L(x-y)(x-y)×F(x-y)+(1-α(x-y))×B(x-y), where L(x,y) denotes the input image pixel value and x and y the image coordinates. The terms F(x.y) and B(x.y) denote the pixel values in the foreground and background, respectively, and α(x.y) denotes the average opacity value of the foreground pixels.

To minimize the inherent global angle deviations due to manual tissue sectioning, the longest horizontal and vertical diameters of each image were first determined; the intersection point was considered as the object center, which was then used to calculate the displacement between two objects. A rigid similarity transformation,[10] appropriate for whole-slide image registration owing to its low computation complexity, was applied to rescale the inconsistent objects and calibrate the angle differences between each HR0 and HTn. The component matrix, which included rotation, translation, and scaling, is described as follows:



Where dx and dy denote translation parameters, S a resale parameter with rotation matrix θ, X and Y the original coordinates of HR0, and X' and Y' the transformed coordinates of HTn.

Nonlinear registration with low resolution images

The global calibration corrected most angle deviations due to manual sectioning, and it was followed by nonlinear image registration to align the inconsistent areas between major tissue elements.[11] A patch-wise image registration model[3] was adapted to efficiently discretize the deformation and evaluate the similarity between LR0 and LTn. To compute the deformation matrix Dy from each aligned LR0/LTn, an objective similarity function with respect to the discretized deformation J (LT0, LRn, Dy) was defined as the computation of

, where the D function measured the image similarity, and the S function measured the reasonability of the transformation.

To assess the similarity, which described the correspondence between neighbor images, the normalized gradient field[12] formula was applied, taking advantage of its intensity-gradient-driven algorithm that could accurately address the registration of differently stained IHC images. The D function that measured the image similarity was defined as follows:



Where D denotes the prolongation operator that interpolates the low-resolution deformation onto the image grid, and

is to assess image similarity. The square brackets in the expression y(x) denote the bilinear interpolation of y based on the four neighboring pixels on the grid of y.

A diffusive regularizer[13] was included to physically define the deformation model during image registration to ensure the robustness and computation speed. To avoid the local minimal value and optimize the joint objective function, the L-BFGS[14] was applied as an optimizer using a discretized analytic derivative and was initialized with the analytic Hessian of the regularizer, which can be interpreted as a special case of linear elasticity as follows:



Multidimensional graph-based registration

To address the minor local deformations of the detailed internal tissue structures generated between differently stained images, a multidimensional graph-based registration was proposed. The algorithm featured two stages: Registration by feature point aggregation and graph-based registration. The high-resolution slides (H) were partitioned into patches first, each of size 4915 × 4915 pixels, with overlapping areas up to 10% of the patch size. For each patch, an individualized transformation was independently performed using a deformation matrix Dy. High-dimensional vectors generated from NGF deformation were obtained, and a cubic B-spline transform was estimated using regularized least-squares error minimization. Finally, the step was refined by resolution progression from low (LTn) to high (HTn).

Registration by feature point aggregation

The pixels in the RGB color space were first translated into a gradient vector image, ΔGi, which was defined as

, where G2x and G2y denote the gradient translations of the horizontal and vertical axes, respectively. The properties of the gradient magnitude contributed to the preservation of differently stained tissue structures, and they clearly demonstrated the differences and similarities in pixel intensity from different gradient vector directions, thereby facilitating the extraction of correct matching feature points. K-means clustering[15] was applied to each translated patch for accurate feature point extraction in discretized-intensity categories.

Matching feature points were extracted using speeded up robust features,[16] which contained steps including matching, feature point detection, and local neighborhood description. A Hessian matrix-based blob detector was used for detecting the feature points. The determinant of the Hessian matrix was applied as a measure of the local variety around the points, while the points were selected when the determinant was maximal. The detector of the feature points contained an indicator to assign invariability for every interest point for a local neighborhood description. These descriptors were adapted to search for correlations between the original image and transformed image when deformations occurred. By comparing the descriptors obtained from different images, matching pairs could be found.

Moreover, the process was refined by boundary control to compute the relative distance and orientation of the vector pointing from the centroid to boundary control points. The boundary control points were set along the object boundary, which was the result of foreground object segmentation, in a clockwise sequence 76 pixels apart. In each alignment after the feature point extraction, the distances of the matching feature points, >100 pixels and <15 pixels, were removed to enhance the significant feature points for the construction of the deformation graph model.

Deformation graph model construction

Nonlinear registration on the patched sections was performed using the deformation graph model construction to reduce the local deformation. The algorithm contained three stages: Deformation graph model construction, graph-based registration, and fusion of deformed patches. For the construction of the deformation graph model, Delaunay triangulation[17] was used to define the deformation spatial relations by feature points. The entire patched image was covered with triangles, and any vertex of each triangle did not fall in the circumscribed circle of other triangles, thereby generating uniqueness. In each align, when the feature points were altered, only a part of the triangular structures needed to be changed, thereby enhancing the computation performance. [Figure 3] shows the graphs of deformation spatial relations from a part of differently stained images.
Figure 3: Example graphs of deformation spatial relations with (a) p63 and (b) EMA. Delaunay triangulation was used to define the deformation spatial relations by feature points. The entire patched image was covered with triangles, and any vertex of each triangle did not fall in the circumscribed circle of other triangles, thereby generating uniqueness

Click here to view


After model construction, the one-to-one corresponding relationship between the pairs of triangle in two aligned images was defined for achieving smooth graph-based registration. [Figure 4] shows an example of alignment; the triangle of moving 1s in the transformation patch was always mapped to the triangle of fixed 1s in the reference patch. All the pairs of the corresponding triangles were transformed using an affine transformation matrix[18] for matching. The transformation matrix AT was defined as follows: AT = Mtranslation × M<scaling × Mrotation × Mshear, which contains translation, scaling, rotation, and shear mapping, where

Figure 4: Process of alignment by the deformation spatial relations graphs of the proposed multidimensional graph-based registration

Click here to view


Finally, to obtain a global smooth deformation, bilinear interpolation was used to fuse the deformed patches into a complete graph. The global smoothness of the fused deformation was ensured by comparing the differences between the two deformation vectors in the overlapping areas.

All the experiments were conducted using MATLAB 2019b together with the C++ open source library. The hardware environments were Intel Core i7-8700K 4.60 GHz processor, 32 G of random access memory at 2400 MHz, and Microsoft Windows 10 operating system. For validating the effectiveness of the proposed method, the average (Avg) and standard deviations (SD) of the Hausdorff distance[3],[19] were calculated. The performance accuracy of the registration was evaluated by calculating the differences between the manual segmentations of the corresponding structures after registration.[1],[3] To assess the accuracy and precision of each alignment, thirty structure objects were manually segmented per slide pair by a pathology expert without the knowledge of the registration result to calculate the Hausdorff distance in a stack of every 5 different IHC stainings (HR0 and HT1-4, total 120 segmentation objects). To assess the inherent error (baseline difference) in serial sectioning, a 4-slide stack of the same CK5/6 staining was also created. The average Hausdorff distances were calculated between slides that are 1 section apart (4 μm in distance), 2 sections apart (8 μm in distance), and 3 sections apart (12 μm in distance).


   Results Top


[Figure 5] shows an example of aligning paired patch images at different stages: Original offset, initial alignment using a published protocol (patch-based nonlinear image registration),[3] and using our proposed method. The aligning precision and accuracy were gradually improved. In the original offset and initial alignment, macrostructural image deviations were noted. Using the published protocol, most macrostructures were aligned, although microstructures (e.g., a hair follicle, as marked in the figure) were not aligned. Using the proposed method, most microstructures were aligned. The results demonstrated that smooth microstructural correspondences were established.
Figure 5: Comparing different methods in terms of the alignment accuracy and precision (10×). (a-d) Results of the original offset, with significant macrostructural alignment deviations were noted (red asterisks). (e-h) Registration results of the initial alignment. (i-l) Registration results obtained by applying patch-based nonlinear region. (m-p) Registration results obtained by applying the proposed method. The alignment accuracy and precision were gradually improved. Within the red circles are the examples of a hair follicle microstructure, showing the best alignment by the proposed method

Click here to view


[Figure 6] shows the examples of feature point extraction in different IHC-stained patch images obtained after performing the proposed registration feature point aggregation at the multidimensional graph-based registration stage. The results demonstrate that the coordinates of most of the important microstructures (determined by a pathologist) were featured during the registration with correspondence.
Figure 6: Registration feature point aggregation. Four paired patch images with feature points were demonstrated. (a and b) p63-K5/6, (c and d) p63-EMA, (e and f) p63-CK7, and (g and h) p63-HER2

Click here to view


To demonstrate the effectiveness of the proposed method in the practical screening of the proteome topology, a pseudocolor system was applied for visualization. Each color represented a different protein highlighted by the corresponding IHC antibodies. The proposed method showed the best alignment through which the protein–protein coexpression or mutually exclusive expression could clearly be elucidated.

The computation time was in average 300 s per patch of the proposed method, as compared with 320 s per patch for the published patch-based nonlinear registration method. It took about 5 h to align two whole-slide images with 60 patches and about 8.3 h to align two images with 100 patches.

The average baseline differences (inherent error caused by serial sectioning) are listed in [Table 1]. The results estimated the minimum Hausdorff distance that can be achieved. The different distances between the sections did not generate a statistically significant (P > 005) deviation from the HR0 slide in Hausdorff distance calculation.
Table 1: Average baseline differences in serial sections of the same immunohistochemical staining with 1 section apart, 2 sections apart, and 3 sections apart by the proposed method

Click here to view


Overall the proposed multidimensional graph-based registration method improved the aligning efficiency and precision [Table 2]. The average Hausdorff distances across a stack of five different IHC stainings (HR0 and HT1-4, total 120 segmentation objects) were 361.79 μm (range: 139.28–688.702 μm), 93.89 μm (range: 59.59–153.85 μm), and 48.93 μm (range: 43.75–54.77 μm) for initial alignment, published patch-based nonlinear registration method, and proposed multidimensional graph-based registration method, respectively. The average standard deviations were 62.82 μm, 50.85 μm, and 14.94 μm, respectively. The best alignment occurred in skin tissue using the antibodies of p63, CK7, and CK5/6 (mean Hausdorff distance 46.99 μm), while the wrost alignment occurred in prostatic tissue stained with MAOa, where the alignment failed [mean Hausdorff distance >1000 μm, or Hausdorff distance uncalculable, [Supplemental Figure 2], and the data were excluded (see Discussion).
Table 2: Performance evaluation of image registration by Hausdorff distances on a stack of 5 serial, whole-slide, immunohistochemical tissue sections by different protocols

Click here to view




   Discussion Top


The precise alignment and registration of consecutive whole-slide histopathological section images is a key step leading to the future topographical proteomics and 3D reconstruction of disease tissues at the cell-level resolution. However, only few models and algorithms, including unsupervised content classification-based nonrigid registration and patch-based nonlinear image registration, were established for this purpose.[3],[4],[6],[20] These methods could be programmed and automatized, providing significant improvements compared with the conventional outline-based or manual-based image overlaying systems. However, in daily laboratory practice, minor issues were still witnessed, one of which was in the application of topographical proteomics. In addition, the alignment between the series of strongly and differently stained whole-slide IHC images was usually a problem. The conventional hematoxylin- and eosin stain or special stains (such as Periodic acid–Schiff stain and Giemsa stain) were global stains, i.e., the dyes stained all the tissue elements by chemical reactions, although certain stains would highlight certain tissue structures. Under these circumstances, pattern recognition can be used for image alignment and registration.[7],[8],[9] IHC stains, however, were nonglobal stains. The principle of IHC stains was the antigen–antibody reaction, and thus, only cells that contained certain proteins could be stained. Although there existed a hematoxylin counterstain in the background, when the stained protein was highly expressed, the background hematoxylin stain would be masked and pose a problem when aligning the neighborhood IHC images, because they were usually mistaken as different objects. In this study, a multidimensional graph-based registration algorithm improved the unsupervised content classification-based nonrigid registration method to address this issue. Our proposed algorithm offered the advantages of rigid and nonrigid registration methods. Initializing the registration of high-resolution slides using a rigid method preserves the main morphology information and providing a deformation matrix generated from nonlinear low-resolution slide registration for following multidimensional graph-based registration provides high-resolution patch images for the local microstructural alignment.

By calculating the Hausdorff distances [see [Table 1], we proved that the proposed method could improve the overall alignment accuracy (represented by the average Hausdorff distances) and precision (represented by the standard deviation of the Hausdorff distances). The published patch-based nonlinear registration method,[3] despite showing a significant improvement from the initial alignment, encountered problems in capturing the specific matching feature points in most local tissue microstructures, because the stained protein distribution and intensity were different in each IHC slide. The proposed graph-based deformation model bypassed this problem through intensity-independent feature point caption and construction, thereby performing effectively. The overall performance improvement was 52.1% and 29.3% for the average and SD values with the Hausdorff distance, respectively.

However, the main limitation of the proposed method is the inherent tissue deviation due to manual serial sectioning. The neighboring two sections, in nature, are significantly alike but not the same. The average human cell size is approximately in the range of 15–20 μm, and using the average and SD deviations of the Housdorff distances (49 and 15 μm, respectively), a deviation of three cells in accuracy and of one cell in precision can be expected. Fortunately, in real human biology, cells with similar or related functions usually work together as microunits, such as in a microvessel, nerve, or folliculosebaceous unit, which has a size >50 μm; additionally, with the microstructure aligned, most functional topology proteomics can be correctly interpreted [see [Figure 5] and [Figure 7]. Most of the human biology works through functional microunits rather than individual cells. Some other modern methods, including next-generation immunohistochemistry using mass cytometry,[21],[22] multichannel or multiplex immunofluorescence,[23],[24],[25] or thick-section confocal microscopy,[26] although could detect multiple protein statuses on a single slide in identical cells giving a highly accurate evaluation on protein interactions, these methods are usually expensive, labor intensive, limited in the interpreted section size, require cell labeling, and demand high techniques, thereby making them unlikely to be daily screening tools. Interestingly, these methods were developed because in those days “the conventional serial-section approach may have instead approximated the cellular co-localization poorly owing to heterogeneity effects.”[25] Now with our proposed method, serial whole-slide IHC sections can be accurately and precisely aligned. The “old-day” approach offers several clear advantages, especially for screening purposes. First, whole-slide images can be reviewed in a bright room (without requiring a dark room for immunofluorescence), providing an overall view of the proteome topology and facilitating pattern recognition. Second, for each protein, the staining condition can be optimized because the stain works on independent slides. By contrast, there is usually a compromise in the staining condition when multiple labels or stains work on the same slide. Third, most pathologists worldwide are acquainted with the conventional IHC images, thereby making the application of this method highly practical from the daily usage viewpoint. Last, because IHC is an essential part in daily pathology practice, even a local hospital can provide all the needed instruments and slides for the proposed method. No new instrument, reagent, or technique is required.
Figure 7: Pseudocolor system demonstrating the immunohistochemical protein expression for practical proteome topology: p63 (red), CK5/6 (green), EMA (blue), and CK7 (yellow) and HER2 (magenta) in protein staining (a-d) and background staining (e-h) areas. The stained areas in each whole-slide immunohistochemical image were extracted and transformed to a pseudocolor. The protein–protein coexpression or mutually exclusive expression could clearly be observed

Click here to view


The alignment quality, however, still heavily depends on the quality of the antibody used. In this project, the best alignment came from skin tissue sections stained with p63, CK7, and CK5/6. These three antibodies gave highly specific and clear staining signals with clean background and minimal cross reaction, thus preserving the skin internal microstructures. By contrast, the alignment failed in prostatic tissue sections stained with MAOa. The antibody had intrinsic poor quality, giving a blurred and nonspecific staining pattern with high background, thus masking the underlying tissue microstructures.

Another limitation of the proposed method is that its ability to treat extreme tissue torsion is still limited, as with other aligning methods.[4] Tissue information based on consistent spatial correlations may be incorrect because of the artifacts introduced by cutting, folding, stretching, or tearing during tissue slicing. Excessive extraction of registration feature points is possible when artifacts are introduced so that mutually exclusive proteins are brought together with variable overlaps. Additionally, the proposed method, although provides accurate and precise image alignment, show no apparent advance in terms of the computation time (average 300 s per patch, as compared with 320 s per patch for the published patch-based nonlinear registration method). For screening purposes, the scanning resolution could be decreased to reduce the computation time.


   Conclusions Top


This study presented a three-step image-registration method to efficiently align serial, whole-slide IHC tissue section images. First, rigid registration was applied for the initial global alignment. Second, the patch-based nonlinear registration method was employed to perform low-resolution registration to achieve alignment between major tissue structures. Third, the proposed multidimensional graph-based registration, combined with a deformation matrix from patch-based nonlinear registration, was used to align the global and local deformation microstructures, improving the registration performance and robustness. Owing to its low cost and minimal requirements, our proposed method offers great advantages in screening the proteome topology in the daily laboratory practice.

Financial support and sponsorship

This work was supported by the Ministry of Science and Technology of Taiwan (grant numbers 105-2633-B-002-006- to C.W.L); National Taiwan University Hospital [grant number 105-M3314 to C.W.L]; Taipei Institute of Pathology (TIP-109-004 to C.W.L).

Conflicts of interest

None declared.



 
   References Top

1.
Lotz J, Berger J, Muller B, Breuhahn K, Grabe N, Heldmann S, et al. Zooming in: High resolution 3D reconstruction of differently stained histological whole slide images. Proc SPIE 2014;9041:904104-1-7.  Back to cited text no. 1
    
2.
Magee D, Song Y, Gilbert S, Roberts N, Wijayathunga N, Wilcox R, et al. Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis. J Pathol Inform 2015;6:6.  Back to cited text no. 2
[PUBMED]  [Full text]  
3.
Lotz J, Olesch J, Muller B, Polzin T, Galuschka P, Lotz JM, et al. Patch-based nonlinear image registration for gigapixel whole slide images. IEEE Trans Biomed Eng 2016;63:1812-9.  Back to cited text no. 3
    
4.
Wang CW, Ka SM, Chen A. Robust image registration of biological microscopic images. Sci Rep 2014;4:6050.  Back to cited text no. 4
    
5.
Roberts N, Magee D, Song Y, Brabazon K, Shires M, Crellin D, et al. Toward routine use of 3D histopathology as a research tool. Am J Pathol 2012;180:1835-42.  Back to cited text no. 5
    
6.
Song Y, Treanor D, Bulpitt AJ, Wijayathunga N, Roberts N, Wilcox R, et al. Unsupervised content classification based nonrigid registration of differently stained histology images. IEEE Trans Biomed Eng 2014;61:96-108.  Back to cited text no. 6
    
7.
Reinhard E, Adhikhmin M, Gooch B, Shirley P. Color transfer between images. IEEE Comput Graph Appl 2001;21:34-41.  Back to cited text no. 7
    
8.
Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 1979;9:62-6.  Back to cited text no. 8
    
9.
Levin A, Lischinski D, Weiss Y. A closed-form solution to natural image matting. IEEE Trans Pattern Anal Mach Intell 2008;30:228-42.  Back to cited text no. 9
    
10.
Brown LG. A survey of image registration techniques. ACM Comput Surv 1992;24:325-76.  Back to cited text no. 10
    
11.
Li X, Orchard MT. New edge-directed interpolation. IEEE Trans Image Process 2001;10:1521-7.  Back to cited text no. 11
    
12.
Haber E, Modersitzki J, editors. Intensity Gradient based Registration and Fusion of Multi-Modal Images. Medical Image Computing and Computer-Assisted Intervention – MICCAI 20; 06. Berlin, Heidelberg: Springer Berlin Heidelberg; 2006.  Back to cited text no. 12
    
13.
Fischer B, Modersitzki J. Fast diffusion registration. Contemp Math 2002;313:117-28.  Back to cited text no. 13
    
14.
Liu DC, Nocedal J. On the limited memory BFGS method for large scale optimization. Math Program 1989;45:503-28.  Back to cited text no. 14
    
15.
Likas A, Vlassis N, Verbeek JJ. The global k-means clustering algorithm. Pattern Recognit 2003;36:451-61.  Back to cited text no. 15
    
16.
Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded-up robust features (SURF). Comput Vision Image Underst 2008;110:346-59.  Back to cited text no. 16
    
17.
Lertrattanapanich S, Bose NK. High resolution image formation from low resolution frames using Delaunay triangulation. IEEE Trans Image Process 2002;11:1427-41.  Back to cited text no. 17
    
18.
Nomizu K, Katsumi N, Sasaki T. Affine Differential Geometry: Geometry of Affine Immersions. Cambridge, England: Cambridge University Press; 1994.  Back to cited text no. 18
    
19.
Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the hausdorff distance. IEEE Trans Pattern Anal Mach Intell 1993;15:850-63.  Back to cited text no. 19
    
20.
Song Y, Treanor D, Bulpitt AJ, Magee DR. 3D reconstruction of multiple stained histology images. J Pathol Inform 2013;4:S7.  Back to cited text no. 20
[PUBMED]  [Full text]  
21.
Louie KB, Bowen BP, McAlhany S, Huang Y, Price JC, Mao JH, et al. Mass spectrometry imaging for in situ kinetic histochemistry. Sci Rep 2013;3:1656.  Back to cited text no. 21
    
22.
Giesen C, Wang HA, Schapiro D, Zivanovic N, Jacobs A, Hattendorf B, et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat Methods 2014;11:417-22.  Back to cited text no. 22
    
23.
Schubert W, Bonnekoh B, Pommer AJ, Philipsen L, Böckelmann R, Malykh Y, et al. Analyzing proteome topology and function by automated multidimensional fluorescence microscopy. Nat Biotechnol 2006;24:1270-8.  Back to cited text no. 23
    
24.
Gerdes MJ, Sevinsky CJ, Sood A, Adak S, Bello MO, Bordwell A, et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc Natl Acad Sci U S A 2013;110:11982-7.  Back to cited text no. 24
    
25.
Clarke GM, Zubovits JT, Shaikh KA, Wang D, Dinn SR, Corwin AD, et al. A novel, automated technology for multiplex biomarker imaging and application to breast cancer. Histopathology 2014;64:242-55.  Back to cited text no. 25
    
26.
Liu YA, Chung YC, Shen MY, Pan ST, Kuo CW, Peng SJ, et al. Perivascular interstitial cells of cajal in human colon. Cell Mol Gastroenterol Hepatol 2015;1:102-19.  Back to cited text no. 26
    


    Figures

  [Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7]
 
 
    Tables

  [Table 1], [Table 2]



 

 
Top
  

    

 
  Search
 
   Browse articles
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
   Introduction
    Materials and Me...
   Results
   Discussion
   Conclusions
    References
    Article Figures
    Article Tables

 Article Access Statistics
    Viewed433    
    Printed8    
    Emailed0    
    PDF Downloaded67    
    Comments [Add]    

Recommend this journal