Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 740  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 




 
Table of Contents    
SYMPOSIUM - ORIGINAL RESEARCH
J Pathol Inform 2013,  4:7

3D reconstruction of multiple stained histology images


School of Computing, University of Leeds, Leeds, United Kingdom

Date of Submission21-Jan-2013
Date of Acceptance21-Jan-2013
Date of Web Publication30-Mar-2013

Correspondence Address:
Yi Song
School of Computing, University of Leeds, Leeds
United Kingdom
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/2153-3539.109864

Rights and Permissions
   Abstract 

Context: Three dimensional (3D) tissue reconstructions from the histology images with different stains allows the spatial alignment of structural and functional elements highlighted by different stains for quantitative study of many physiological and pathological phenomena. This has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, and enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications). Methods: This paper evaluates three strategies for 3D reconstruction from sets of two dimensional (2D) histological sections with different stains, by combining methods of 2D multi-stain registration and 3D volumetric reconstruction from same stain sections. Setting and Design: The different strategies have been evaluated on two liver specimens (80 sections in total) stained with Hematoxylin and Eosin (H and E), Sirius Red, and Cytokeratin (CK) 7. Results and Conclusion: A strategy of using multi-stain registration to align images of a second stain to a volume reconstructed by same-stain registration results in the lowest overall error, although an interlaced image registration approach may be more robust to poor section quality.

Keywords: 3D reconstruction, computerized diagnosis, digital pathology, histological image, image analysis, multi-stain registration


How to cite this article:
Song Y, Treanor D, Bulpitt AJ, Magee DR. 3D reconstruction of multiple stained histology images. J Pathol Inform 2013;4:7

How to cite this URL:
Song Y, Treanor D, Bulpitt AJ, Magee DR. 3D reconstruction of multiple stained histology images. J Pathol Inform [serial online] 2013 [cited 2019 Dec 7];4:7. Available from: http://www.jpathinformatics.org/text.asp?2013/4/2/7/109864


   Introduction Top


Histopathology is the microscopic examination of tissue sections, which are colored with different stains to display different functional or physical properties. Three dimensional (3D) tissue reconstruction from the digital images of serial sections has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications). This can be obtained by successively applying two dimensional (2D) image-to-image registrations and then concatenating the set of aligned images to form a 3D volumetric dataset. Various methods based on this idea have been presented for 3D reconstruction from a set of images of same-stain serial sections. [1],[2],[3],[4] Recently, there is increasing research interest in registering 2D histology images of consecutive sections with different stains, [5],[6],[7],[8] , since automatic co-registration of these images allows the spatial alignment of structural and functional elements in different modalities for quantitative study of many physiological and pathological phenomena. However, there is no work known to the authors addressing the problem of 3D reconstruction from images of sections with different stains. In this paper, we discuss three strategies for combining the solutions of automatic 2D multi-stain registration and 3D volumetric reconstruction from same-stain sections. As the 2D multi-stain registration and same-stain volumetric reconstruction are the key steps of all 3D reconstruction strategies, for the completeness, we provide the necessary details about our implementations of 2D registrations in section Methods.


   Materials Top


Two human liver surgical Specimens (A, B) are used to examine three 3D reconstruction strategies. Local NHS research ethical approval was obtained for use of the human tissue. On Specimen A, four artificial vertical holes were made for the quantitative evaluation of the accuracy of the 3D reconstruction. Specimen B was chosen because there are relatively consistent anatomic features present in the stack of sections. Using standard histological techniques, the specimens were formalin fixed, paraffin embedded, and cut into ~5μm sections using a microtome.

  • For Specimen A, every 4 th section was selected and stained with H and E; every 5 th section was stained with Sirius Red. Twenty five sections of each stain were prepared, with 50 sections in total.
  • For Specimen B, every 2 nd , 3 rd , 4 th section was stained with H and E while every 6 th , 7 th , 8 th sections were stained with CK7. Fifteen sections of each stain were prepared, with 30 sections in total.
Sections were scanned at 20× magnification with average image size ~1.4 G (Specimen A) and 40× with average image size ~18.5 G (specimen B) using an Aperio XT scanner. To verify the influence of misaligned 2D images on the various 3D reconstruction strategies, images of sections with the following problems are included:

  • Luminance gradient: Sections mounted close to the edge of the glass slide produce images with significant luminance gradients [Figure 1]-A1, [Figure 1]-A2.
  • Non-tissue noise: Dust and air bubbles in the slide [Figure 1]-A1, [Figure 1]-A2.
  • Damaged and missing sections: During sectioning and mounting, sections are occasionally torn, folded [Figure 1]-B1.
  • Staining variations: Differences in section thickness, staining duration, and stain concentration result in colour variations [Figure 1]-B2.
    Figure 1: Example histological sections on which the green marks are annotations used for the evaluation

    Click here to view

   Methods Top


There are several potential approaches for 3D reconstruction from images of differently stained sections which we have implemented and evaluated. An intuitive option is to apply 2D registration sequentially for each stain separately (i.e. same-stain registration) after the reference image (i.e. the first section) for each stack has been aligned by applying 2D multi-stain registration [[Figure 2], Strategy 1]. The second solution is to apply same-stain registration on the stack of images with one stain (e.g. T1) sequentially and then aligning every image in the stack of the other stain (e.g. T2) to its adjacent image in the stack of stain T1 using multi-stain registration [[Figure 2], Strategy 2]. The third approach is interlacing images of stain T1 and T2 in the same sequence as sectioning and applying multi-stain registration for all neighboring images [[Figure 2], Strategy 3]. In "Results" section, we present a comparison study of these options. The 3D registration approaches discussed in this paper are the extension of: 1) Our previous works on 3D volume reconstruction from 2D registration of images of same-stain serial sections [1] and 2) our unsupervised content classification based on 2D multi-stain registration method (manuscript under preparation). To make this paper independently readable, we briefly review our 2D registration methods in the following sections.
Figure 2: Strategies of 3D registration of multi-stain pathological images

Click here to view


Automatic 2D:2D Same-Stain Registration of Histology Images

Registration consists in two main steps: 1) Initial rigid alignment of an red, green, blue (RGB) image pair at low resolution X0 using a phase correlation based method (a combination of [9] and [10] to recover scale and rotation) applied to grayscale image; 2) Non-rigid alignment is done by dividing each image into small patches (from our cross-validation study, the best results are given at 256 × 256 pixels per patch) and computing the rigid transformation of these patches separately by applying the phase correlation based method. The deformation field is then represented by five vectors for each patch, i.e. one at each corner and one at the centre. A cubic B-Spline transform is estimated to approximate the deformation field using regularized least squares error minimization. [11] The non-rigid transform in Step 2 is refined by a multi-resolution (from coarse X1 to fine Xn ) approach. The patch matching approach is the key to tackle large image size (gigabytes at high resolution).

Automatic 2D:2D Multi-Stain Registration of Histology Images

A challenge when registering histology images with different stains is that different stains highlight different substances in tissues, resulting in dissimilar structural appearances on adjacent tissue sections. We address this challenge by proposing an unsupervised content classification method, which automatically identifies common content classes from differently stained histology image pair. This method creates two multi- channel probability images [Figure 3]c for the non-rigid alignment, as in step 2 where the grayscale images are used for the same-stain registration. The workflow of multi-stain registration is illustrated in [Figure 3], and the results at resolution X i (1 ≤ in) of each step are exemplified on the right hand side. The content classification algorithm is summarized in [Table 1], where we assume the input image pair at resolution X i has been pre-registered by the transformation estimated at resolution X i-1 (within a multi-resolution framework).
Figure 3: Workflow of unsupervised content classification based registration of multi-stain histology images. Examples results of each step are illustrated at the right side. (a) Original tissue sections stained with H and E and Sirius Red stains. (b) Rigidly aligned image pair. (c) Example output of three channel probability image pair. Each channel is corresponding to a content class automatically identified from the image pair in (a). (d) Non-rigid registration result

Click here to view
Table 1: Content classification algorithm

Click here to view


Spatial location features are introduced as complementary to the appearance features. This overcomes the problem where the appearance features alone (for some stains) are not sufficient to separate certain image regions, which have similar appearance, but belong to different content classes. As that the content classes are emergent from co-occurrence statistics between the image pairs, they do not necessarily resemble to real anatomic classes. Note this is not a prerequisite for registration.


   Results Top


Evaluation on 2D Multi-Stain Registration

The 2D unsupervised content classification-based registration method plays a key role in our 3D reconstruction study. Thus, we evaluate the 2D multi-stain registration method using three histological image pairs with six different stains from a liver specimen, a intervertebral disc specimen, and a colon specimen. We manually annotate 20~30 regions (the number of pixels per annotation ranges 2000~5600) on each image pair. The alignment accuracy for each image pair is evaluated by measuring Hausdorff distance (μm) of corresponding annotations at full-resolution level (at which the image was originally scanned). The boxplot of registration accuracy is shown in [Figure 4].
Figure 4: Boxplot of the alignment accuracy. x-axis indicates the annotations. y-axis is the Hausdorff distance (μm). The solid lines indicate the registration results with using content classification method. The dash lines indicate the results without using content classification method. (a) Colon specimen (H and E, Cytokeratin Immunohistochemical). (b) Vertebral specimen (Alcian Blue, EPSR). (c) Liver specimen (Reticulin, Masson trichrome)

Click here to view


Evaluation of 3D Reconstruction Strategies

[Figure 5] illustrates the influence of different 3D reconstruction strategies (described in "Materials and Methods" section) on the reconstructed specimens by stacking all the registered images of stain A serially as when they were cut. For example, for the reconstructed Specimen A, the odd number images are those stained with H and E and the even number images are stained with Sirius Red. The Hausdorff distances (μm) are measured at 20× magnification for Specimen A. The graphs at the position n depict the distance between the n+1 th images and the 1 st image (i.e. accumulated error). Each line plotted in the graph corresponds to one manual annotation, as shown in [Figure 1].
Figure 5: (a-c) Accumulated Hausdorff distances of reconstructed Specimen A by applying Strategy 1, 2, 3, respectively. x-axis is series number of sections. (d) Boxplot of the Hausdorff distances between successive images in each stack

Click here to view


[Figure 5]d (Strategy 1) shows large misalignments exist between successive images of different stains, due to the fact that Strategy1 is based on two sets of independent same-stain registrations. This results in a large accumulated error with respect to the first image [Figure 5]a. Strategy 2 shows better registration accuracy between stains [Figure 5]d resulted from applying same-stain registration on odd number images (H and E stain) followed by multi-stain registration to align every even number images (Sirius Red stain) to its neighboring H and E image. There is also less accumulation of errors with respect to the first image [Figure 5]b. [Figure 5]d illustrates the Strategy 3 also has good registration accuracy between successive images of different stains; however, accumulated error is larger than Strategy 2 [Figure 5]c.

Unlike Specimen A, there are no artificial markers on the Specimen B. As the influence of anatomic changes overrides the registration errors, for Specimen B, we do not provide Hausdorff evaluation based on annotations of real anatomic structures. Instead, the smoothness of the reconstructed specimens in coronal views offers a more intuitive qualitative way of comparing the three strategies, as in [Figure 6].
Figure 6: Comparison of 3D volumetric reconstructions of Specimen A, B with different strategies. The coronal views of the reconstructed Specimen A (Top). View plane locates at annotations a1, a2. The Coronal views of the reconstructed Specimen B (Bottom). View plane locates at annotations b2

Click here to view


[Figure 6] shows a coronal view of the reconstructed specimens. For Specimen A (top), Strategy 2 gives the smoothest 3D reconstruction, however, several outliers exist. Strategy 1 has a smooth reconstruction before the first big misalignment presents, which leads to the poor reconstruction of the rest part of the specimen. Although Strategy 3 has the smallest Hausdorff distance between successive images, as illustrated in [Figure 5]d, the reconstructed volume is not as smooth as Strategy 2, reflecting the fact that Strategy 3 has random accumulated registration errors. For Specimen B (bottom), similar conclusions can be drawn. Strategy 1 has larger registration error at the end of the reconstructed stack than other two strategies. Strategy 2 and 3 give much smoother blood vessel walls and specimen boundary, and their difference is minor in this case.

In addition to the registration accuracy, computational efficiency is also an important factor to be considered. [Table 2] exemplifies the execution times of same-stain and multi-stain 2D registrations at different resolutions. All computations were carried out on a desktop computer with the following specifications: Intel i7 dual core ×3.07 GHz, 8 GB RAM, Windows 7. The code is not yet optimized for parallel computation.
Table 2: Computation efficiency

Click here to view



   Conclusion Top


In our experiments, we also notice that the multi-stain 2D registration method is less sensitive to problematic sections and performs accurately on serial sections of continuous cut. However, when the gap between the two sections increases (i.e. more sections are left out of the reconstruction), the registration accuracy correspondingly decreases since larger variations in tissue structures are present between the two sections, and thus affects the result of co-occurrence statistics, and, thus, unsupervised classification performance. In summary, Strategy 3 (only involving multi-stain 2D registration) has an advantage in dealing with some problematic sections and provides good 3D reconstruction results (when only few sections in the cut are left out of 3D reconstruction), however, it requires 2×n-1 multi-stain registrations (n the number of images in each stain). Strategy 2 results in the lowest accumulated errors if those problematic sections can be discarded before the registration, but is more sensitive to these problematic sections. Moreover, it only requires n-1 same-stain 2D registrations plus n multi-stain 2D registrations, which is much more computationally efficient than Strategy 3. Both Strategy 2 and Strategy 3 are practically usable.

 
   References Top

1.Roberts N, Magee D, Song Y, Brabazon K, Shires M, Crellin D, et al. Towards routine use of 3D histopathology as a research tool. American Journal of Pathology, 2012;180:1835-1842.  Back to cited text no. 1
    
2.Kurien T, Boyce RW, Paish EC, Ronan J, Maddison J, Rakha EA, et al. Three dimensional reconstruction of a human breast carcinoma using routine laboratory equipment and immunohistochemistry. Journal of Clinical Pathology 2005;58:968-972.  Back to cited text no. 2
    
3.Magee D, Treanor D, Quirke P. A new image registration algorithm with application to 3D histopathology. Proceedings of the 3 rd Medical Image Computing and Computer Assisted Intervention (MICCAI) Workshop on Microscopic Image Analysis with Applications in Biology (MIAAB). New York; September 5-6, 2008.  Back to cited text no. 3
    
4.Pitiot A, Bardinet E, Thompson P, Malandain G. Piecewise affine registration of biological images for volume reconstruction. Medical Image Analysis 2006;10:465-483.  Back to cited text no. 4
    
5.Can A, Bello M, Cline HM, Tao X, Ginty F, Sood A, et al. Multi-modal imaging of histological tissue sections. Biomedical Imaging: From Nano to Macro, 2008. ISBI 2008. 5 th IEEE International Symposium. Paris; May 14-17, 2008.   Back to cited text no. 5
    
6.Cooper L, Sertel O, Kong J, Lozanski G, Huang K, Gurcan M. Feature-based registration of histopathology images with different stains: An application for computerized follicular lymphoma prognosis. Computer Methods Programs in Biomedicine. 2009;96:182-92.  Back to cited text no. 6
    
7.Cooper L, Naidu S, Leone G, Saltzl J, Huang K. Registering high resolution microscopic images with different histochemical staining - a tool for mapping gene expression with cellular structures. Proceedings of the 3 rd Medical Image Computing and Computer Assisted Intervention (MICCAI) Workshop on Microscopic Image Analysis with Applications in Biology (MIAAB). Piscataway; September 21, 2007.  Back to cited text no. 7
    
8.Mueller D, Vossen D, Hulsken B. Real-time deformable registration of multi-modal whole slides for digital pathology. Computerized Medical Imaging and Graphics. 2011;35:542-556.  Back to cited text no. 8
    
9.Casasent D, Psaltis D. Position, rotation, and scale invariant optical correlation. Applied Optics 1976;15:1795-1799.  Back to cited text no. 9
    
10.Kuglin C, Hines D. The phase correlation image alignment method. Proceedings of IEEE International Conference Cybernetics and Society; 1975. p. 163-165.  Back to cited text no. 10
    
11.Hansen P. Rank-deficient and discrete ill-posed problems: Numerical aspects of linear inversion. Society for Industrial and Applied Mathematics (SIAM) 1999.  Back to cited text no. 11
    


    Figures

  [Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6]
 
 
    Tables

  [Table 1], [Table 2]


This article has been cited by
1 A Survey of Methods for 3D Histology Reconstruction
Jonas Pichat,Juan Eugenio Iglesias,Tarek Yousry,Sébastien Ourselin,Marc Modat
Medical Image Analysis. 2018;
[Pubmed] | [DOI]
2 Histopathology: ditch the slides, because digital and 3D are on show
Ilaria Jansen,Marit Lucas,C. Dilara Savci-Heijink,Sybren L. Meijer,Henk A. Marquering,Daniel M. de Bruin,Patricia J. Zondervan
World Journal of Urology. 2018;
[Pubmed] | [DOI]
3 Label-free automated three-dimensional imaging of whole organs by microtomy-assisted photoacoustic microscopy
Terence T. W. Wong,Ruiying Zhang,Chi Zhang,Hsun-Chia Hsu,Konstantin I. Maslov,Lidai Wang,Junhui Shi,Ruimin Chen,K. Kirk Shung,Qifa Zhou,Lihong V. Wang
Nature Communications. 2017; 8(1)
[Pubmed] | [DOI]
4 Nephron morphometry in mice and rats using tomographic microscopy
Robyn F. R. Letts,Xiao-Yue Zhai,Charita Bhikha,Birgitte L. Grann,Nicklas B. Blom,Jesper Skovhus Thomsen,David M. Rubin,Erik I. Christensen,Arne Andreasen
American Journal of Physiology-Renal Physiology. 2017; 312(1): F210
[Pubmed] | [DOI]
5 as -PSOCT: Volumetric microscopic imaging of human brain architecture and connectivity
Hui Wang,Caroline Magnain,Ruopeng Wang,Jay Dubb,Ani Varjabedian,Lee S. Tirrell,Allison Stevens,Jean C. Augustinack,Ender Konukoglu,Iman Aganj,Matthew P. Frosch,Jeremy D. Schmahmann,Bruce Fischl,David A. Boas
NeuroImage. 2017;
[Pubmed] | [DOI]
6 Transformation diffusion reconstruction of three-dimensional histology volumes from two-dimensional image stacks
Ramón Casero,Urszula Siedlecka,Elizabeth S. Jones,Lena Gruscheski,Matthew Gibb,Jürgen E. Schneider,Peter Kohl,Vicente Grau
Medical Image Analysis. 2017; 38: 184
[Pubmed] | [DOI]
7 Zonated Quantification of Steatosis in an Entire Mouse Liver
Lars Ole Schwen,André Homeyer,Michael Schwier,Uta Dahmen,Olaf Dirsch,Arne Schenk,Lars Kuepfer,Tobias Preusser,Andrea Schenk
Computers in Biology and Medicine. 2016;
[Pubmed] | [DOI]
8 Serial 3D Imaging Mass Spectrometry at Its Tipping Point
Andrew D. Palmer,Theodore Alexandrov
Analytical Chemistry. 2015; 87(8): 4055
[Pubmed] | [DOI]
9 Seeing Through the Surface: Non-invasive Characterization of Biomaterial–Tissue Interactions Using Photoacoustic Microscopy
Yu Shrike Zhang,Lihong V. Wang,Younan Xia
Annals of Biomedical Engineering. 2015;
[Pubmed] | [DOI]
10 A Method for 3D Histopathology Reconstruction Supporting Mouse Microvasculature Analysis
Yiwen Xu,J. Geoffrey Pickering,Zengxuan Nong,Eli Gibson,John-Michael Arpino,Hao Yin,Aaron D. Ward,Andrew H. Beck
PLOS ONE. 2015; 10(5): e0126817
[Pubmed] | [DOI]
11 Combined use of micro computed tomography and histology to evaluate the regenerative capacity of bone grafting materials
Anja K. Stalder,Bernd Ilgenstein,Natalia Chicherova,Hans Deyhle,Felix Beckmann,Bert Müller,Simone E. Hieber
International Journal of Materials Research. 2014; 105(7): 679
[Pubmed] | [DOI]
12 Optical-Resolution Photoacoustic Microscopy for Volumetric and Spectral Analysis of Histological and Immunochemical Samples
Yu Shrike Zhang,Junjie Yao,Chi Zhang,Lei Li,Lihong V. Wang,Younan Xia
Angewandte Chemie. 2014; 126(31): 8237
[Pubmed] | [DOI]
13 Optical-Resolution Photoacoustic Microscopy for Volumetric and Spectral Analysis of Histological and Immunochemical Samples
Yu Shrike Zhang,Junjie Yao,Chi Zhang,Lei Li,Lihong V. Wang,Younan Xia
Angewandte Chemie International Edition. 2014; 53(31): 8099
[Pubmed] | [DOI]
14 Non-rigid landmark-based large-scale image registration in 3-D reconstruction of mouse and rat kidney nephrons
Yan-Ling Zhang,Shi-Jie Chang,Xiao-Yue Zhai,Jesper Skovhus Thomsen,Erik I. Christensen,Arne Andreasen
Micron. 2014;
[Pubmed] | [DOI]



 

 
Top
  

    

 
  Search
 
   Browse articles
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
   Introduction
   Materials
   Methods
   Results
   Conclusion
    References
    Article Figures
    Article Tables

 Article Access Statistics
    Viewed7081    
    Printed85    
    Emailed1    
    PDF Downloaded1254    
    Comments [Add]    
    Cited by others 14    

Recommend this journal