|J Pathol Inform 2019,
Automated computational detection, quantitation, and mapping of mitosis in whole-slide images for clinically actionable surgical pathology decision support
Munish Puri1, Shelley B Hoover1, Stephen M Hewitt2, Bih-Rong Wei3, Hibret Amare Adissu1, Charles H C Halsey1, Jessica Beck4, Charles Bradley5, Sarah D Cramer6, Amy C Durham5, D Glen Esplin7, Chad Frank8, L Tiffany Lyle9, Lawrence D McGill7, Melissa D Sánchez5, Paula A Schaffer8, Ryan P Traslavina10, Elizabeth Buza5, Howard H Yang1, Maxwell P Lee1, Jennifer E Dwyer1, R Mark Simpson1
1 Laboratory of Cancer Biology and Genetics, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
2 Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
3 Laboratory of Cancer Biology and Genetics, Center for Cancer Research, National Cancer Institute; Frederick National Laboratory for Cancer Research, Leidos Biomedical Research, Inc., MD, USA
4 Laboratory of Human Carcinogenesis, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
5 Department of Pathobiology, University of Pennsylvania, Philadelphia, PA, USA
6 Cancer and Inflammation Program, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
7 Animal Reference Pathology, Salt Lake City, UT, USA
8 Department of Microbiology, Immunology, and Pathology, Veterinary Diagnostic Laboratory, Colorado State University, Fort Collins, CO, USA
9 Women's Malignancies Branch, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
10 Section of Infections of the Nervous System, National Institute of Neurological Disorders and Stroke, Bethesda, MD, USA
|Date of Submission||08-Aug-2018|
|Date of Acceptance||27-Nov-2018|
|Date of Web Publication||07-Feb-2019|
Dr. R Mark Simpson
Laboratory of Cancer Biology and Genetics, Center for Cancer Research, National Cancer Institute, 9000 Rockville Pike, Bethesda, MD 20892
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Background: Determining mitotic index by counting mitotic figures (MFs) microscopically from tumor areas with most abundant MF (hotspots [HS]) produces a prognostically useful tumor grading biomarker. However, interobserver concordance identifying MF and HS can be poorly reproducible. Immunolabeling MF, coupled with computer-automated counting by image analysis, can improve reproducibility. A computational system for obtaining MF values across digitized whole-slide images (WSIs) was sought that would minimize impact of artifacts, generate values clinically relatable to counting ten high-power microscopic fields of view typical in conventional microscopy, and that would reproducibly map HS topography. Materials and Methods: Relatively low-resolution WSI scans (0.50 μm/pixel) were imported in grid-tile format for feature-based MF segmentation, from naturally occurring canine melanomas providing a wide range of proliferative activity. MF feature extraction conformed to anti-phospho-histone H3-immunolabeled mitotic (M) phase cells. Computer vision image processing was established to subtract key artifacts, obtain MF counts, and employ rotationally invariant feature extraction to map MF topography. Results: The automated topometric HS (TMHS) algorithm identified mitotic HS and mapped select tissue tiles with greatest MF counts back onto WSI thumbnail images to plot HS topographically. Influence of dye, pigment, and extraneous structure artifacts was minimized. TMHS diagnostic decision support included image overlay graphics of HS topography, as well as a spreadsheet and plot of tile-based MF count values. TMHS performance was validated examining both mitotic HS counting and mapping functions. Significantly correlated TMHS MF mapping and metrics were demonstrated using repeat analysis with WSI in different orientation (R2 = 0.9916) and by agreement with a pathologist (R2 = 0.8605) as well as through assessment of counting function using an independently tuned object counting algorithm (OCA) (R2 = 0.9482). Limits of agreement analysis support method interchangeability. MF counts obtained led to accurate patient survival prediction in all (n = 30) except one case. By contrast, more variable performance was documented when several pathologists examined similar cases using microscopy (pair-wise correlations, rho range = 0.7597–0.9286). Conclusions: Automated TMHS MF segmentation and feature engineering performance were interchangeable with both observer and OCA in digital mode. Moreover, enhanced HS location accuracy and superior method reproducibility were achieved using the automated TMHS algorithm compared to the current practice employing clinical microscopy.
Keywords: Cancer grading, computer-assisted diagnosis/prognosis, feature engineering, image segmentation, method reproducibility, pathology imaging informatics, proliferation index
|How to cite this article:|
Puri M, Hoover SB, Hewitt SM, Wei BR, Adissu HA, Halsey CH, Beck J, Bradley C, Cramer SD, Durham AC, Esplin D G, Frank C, Lyle L T, McGill LD, Sánchez MD, Schaffer PA, Traslavina RP, Buza E, Yang HH, Lee MP, Dwyer JE, Simpson R M. Automated computational detection, quantitation, and mapping of mitosis in whole-slide images for clinically actionable surgical pathology decision support. J Pathol Inform 2019;10:4
|How to cite this URL:|
Puri M, Hoover SB, Hewitt SM, Wei BR, Adissu HA, Halsey CH, Beck J, Bradley C, Cramer SD, Durham AC, Esplin D G, Frank C, Lyle L T, McGill LD, Sánchez MD, Schaffer PA, Traslavina RP, Buza E, Yang HH, Lee MP, Dwyer JE, Simpson R M. Automated computational detection, quantitation, and mapping of mitosis in whole-slide images for clinically actionable surgical pathology decision support. J Pathol Inform [serial online] 2019 [cited 2020 Jan 18];10:4. Available from: http://www.jpathinformatics.org/text.asp?2019/10/1/4/251845
| Introduction|| |
The proliferative capacity of tumors contributes to their growth and metastasis. Identification and enumeration of mitotic figures (MFs) in cancer biopsy tissue sections microscopically is a surrogate for proliferative activity for many tumor types.,,,,,,,,, Enumeration of MF can be an independent clinically prognostic biomarker relevant for tumor grading. Counting MF on hematoxylin and eosin (H&E)-stained tissue sections, which is not a mitotic-specific stain, is a challenging exercise in MF recognition that requires experience. The strategy can involve selecting tumor areas with the greatest number of MFs, known as mitotic hotspots (HSs). The tedious and time-consuming nature of this error-prone task has been documented in studies examining the reproducibility of MF counting.,,,,,, The variation in obtaining tumor mitotic count values can be attributed to issues such as varying recognition accuracy. For example, agreement about what is a MF and its differentiation from other nuclear structures is not unanimous, and choice of tumor tissue area for evaluation can also play a role in designation of grade., Other limitations such as cellular level variations in color, intensity, and morphological shape/size can contribute to counting errors. These circumstances can result in poor reproducibility and discordant inter- and intra-reader mitotic count values.
Approaches such as antibody immunolabeling of MF and the automation of MF counting through computer-assisted image analysis offer the opportunity to minimize variability and increase reproducibility in assessing tumor proliferative activity in tissue biopsies at the time of diagnosis.,, Molecular markers of mitosis, such as phosphorylated histone H3 (pHH3) and MKI67 (Ki-67), improve prognostic utility of counting MF compared to routine H&E tissue stains.,,,,, Active development of computer-assisted image analysis indicates promise for improving reproducibility and accuracy.,,,,,,,,,, The ability to specifically identify and segment the proliferating cells in digitized whole-slide images (WSIs) using computer-assisted diagnosis (CAD) can be complex, however. The potential to miscategorize MF based on morphological characteristics can become amplified once biopsies are optically scanned for image analysis.
Certain challenges in assessing proliferative activity using computer-assisted decision support have been described, and a limited number of automated approaches for detecting MF in tissue sections have been published.,,,,, Some shortcomings in image processing performance persist, nevertheless. Chief among these includes insufficient locating clinically meaningful mitotic HS regions topographically, substantial minimization of common confounding tissue artifacts, generation of MF counts that are clinically relatable to the MF count values currently produced using conventional microscopy, and the ability for pathologists to quality assure algorithm performance on a patient by patient basis. Development of an automated system improving upon these features to benefit tumor grading based on mitotic activity was a primary objective.
In this study, the automated topometric HS (TMHS) algorithm addresses these needed analytics, providing topographic information on mitotic HS location in the tissue context, while yielding MF metrics as surrogate for the tumor proliferative activity. We developed a feature-based computational approach built upon processing WSI as rendered image grid tiles for extracting immunolabeled MF and incorporated filters capable of subtracting several common tissue artifacts. Performance of the automated image processing characteristics was validated in 30 naturally occurring canine mucosal melanoma tissue biopsies. Naturally occurring canine mucosal melanoma serves as a valuable preclinical model for human mucosal and other triple wild-type melanomas.,, Repeat analysis of these clinical specimens using the automated TMHS mapping algorithm provided significant evidence of method interchangeability with a pathologist or a second algorithm, based on counting MF at the automated TMHS identified HS. Automated TMHS provides needed enhancements for clinically actionable mitotic HS mapping, the prognostic utility of which was significantly correlated with the standard of care.
| Materials and Methods|| |
Formalin-fixed and paraffin-embedded biopsy tissue samples from 30 dogs with naturally occurring mucosal melanoma (manuscript in preparation) were used to evaluate the automated TMHS mitotic mapping application. The biopsy specimens were surgically obtained from dogs in the course of veterinary patient care. The pathological characteristics of human melanoma were shared among these canine melanomas. Patients selected for analysis had follow-up survival data available.
Five-micrometer-thick paraffin-embedded tissue sections that ranged in area from 30.32 to 396.83 mm2 were placed on silanated slides for immunohistochemistry (IHC). Slides were deparaffinized and rehydrated. Following rehydration, slides were bleached in 3% hydrogen peroxide in phosphate-buffered saline (pH 7.2) at 60°C for 30 min to remove melanin. Heat-induced epitope retrieval was performed using citrate solution (S1699, pH 6.0, Agilent Dako, Santa Clara, CA) for 15 min in a steam bath (Black & Decker, Beachwood, OH, USA) and allowed to cool an additional 15 min on the bench at room temperature. Subsequent immunolabeling steps using anti-pHH3, an antibody targeting a mediator of chromatin condensation during cell replication, were performed using an autostainer (Agilent Dako, Santa Clara, CA, USA) at room temperature with an alkaline phosphatase immunoenzyme polymer-linked detection system and permanent red chromogen (K535511-2, Agilent Dako, Santa Clara, CA, USA). Incubation steps included endogenous enzyme block for 10 min, protein block for 20 min, primary antibody (rabbit monoclonal anti-pHH3, ab32107 (clone E173), Abcam, Cambridge, MA, USA) diluted at 1:50 (Dako Antibody Diluent) and incubated for 1 h, followed by polymer-conjugated secondary antibodies as prepared in the kit for 30 min, alkaline phosphatase enzyme with enhancer for 30 min, and permanent red chromogen for 20 min. Slides were then removed from the autostainer, counterstained with hematoxylin, dehydrated, cleared, and coverslipped. Melanomas were processed along with tissue sections of canine intestines, including crypt epithelia, as a positive control tissue. Tissue areas of tumor-adjacent normal and intestinal strata superficial to glandular mucosae served as negative control tissue.
Whole-slide image scanning
Anti-pHH3-immunolabeled slides were scanned at ×20 using an Aperio AT2 digital slide scanner (Leica Biosystems, Buffalo Grove, IL, USA) to create WSI data files at 0.50 μm/pixel resolution. Image files were stored in spectrum image management system and viewed and annotated using Aperio ImageScope Software (Leica Biosystems, Buffalo Grove, IL, USA).
Computational framework for topometric mapping of mitotic hotspots
The computational pipeline is devised and developed in MATLAB image processing and computer vision toolbox (MathWorks, version R2017a, Natick, MA, USA) for WSI of tissue sections. Detection and mapping of MF in cells undergoing mitotic (M) phase is broadly based on color, morphology (shape), and texture (surface roughness) features. Image processing algorithms, including color-based segmentation, color-to-gray conversion, gray-to-binary conversion, counting binary dots, and mapping local features, are used to implement the developed workflow [Figure 1]. Unique coding loops for batch processing image grid tiles and for integrating different building blocks contribute operational sequences. Methods include design of a custom graphical user interface (GUI) to enable downstream unsupervised execution [Supplemental Figure S1 [Additional file 1]].
|Figure 1: Three phases of the analytic workflow for detecting, quantifying, and mapping proliferative cell nuclei (mitotic figure) in tissue whole-slide images. Initial phase involves creating grid tile regions of interest covering the entire tissue and extracting the necessary features to detect and segment phosphorylated histone H3-immunolabeled nuclei, a marker of proliferating cells in mitosis (M phase). In Phase II, the mitotic figure objects of interest are counted, and each grid tile tally is exported to a spreadsheet and rank ordered according to total counts. In the final conceptual Phase III, selected tiles (with greatest counts) are analyzed to obtain local features which can be mapped back to the whole slide image for display|
Click here to view
Image analysis execution encompassed the entire image region using a rendered grid subdivided into 3000 × 3000 pixel tiles, which coincided to image analysis at approximately ×4 magnification, relative to an image viewed using standard light microscopy [Figure 2]a. Each grid tile corresponded to 1.514 mm × 1.514 mm (2.292 mm2) tissue area/tile, which were then segregated into tiles containing tissue images and those lacking tissue. To segment and extract MF in images, a red, green, and blue (RGB) color space was established for pHH3-immunolabeled MF, which enabled detection of the red chromogenic signal developed for the IHC. During MF segmentation, additional color-based segmentation filters, as well as morphology and thresholding filters, were processed to segment MF from the tissue image and to digitally subtract common individual or combinations of shape- and color-based tissue and stain-related artifacts that could confound MF object counting function [Supplemental Figure S2 [Additional file 2]]. Approximately 7% of the 3356 image tiles eventually analyzed were used in training.
|Figure 2: Pictorial depiction of computational workflow for topometric proliferative hotspot detection and mapping for tumor tissue. (a) Input whole-slide image is overlain with a grid of tiles corresponding to 3000 × 3000 pixels each or approximately ×4 magnification. The processing includes a series of steps to reduce the confounding effects of tissue and staining artifacts [Supplemental Figure S2]. Specific color and shape object features corresponding to red chromogen-immunolabeled proliferating cell nuclei within the image are segmented and quantified according to filters established. (b) Mitotic figure extracted features are binarized, counted, and exported to a spreadsheet, as a rank order set of tiles, based on count magnitude. These mitotic figure values are represented in a slope plot for the specimen for up to the first 100 tiles containing the most mitotic figure. (c) Local tissue invariant feature extraction methods are used to obtain the proper orientation of selected identifying features within grid tiles of interest, for those tiles with the largest counts, tiles h1–h5. These are then marked by a bounding box and displayed on the thumbnail image of the input tissue image|
Click here to view
Extracted MFs were converted to gray values based on their respective features. RGB-to-gray function parameters were established to process the true RGB color image to the grayscale intensity image. Segmented MFs were converted to a binary image mask using clustering-based Otsu's method to choose the threshold value for minimizing intraclass variance of the black and white pixels. In succeeding steps, counting algorithms were used to count MF per tile of grid-based image partitioning. MF count data were autoexported to a spreadsheet along with corresponding image tile spatial information. To topographically localize the greatest MF activity areas from the counts generated back onto the tissue, an algorithm (speeded-up robust features [SURF]) was employed to invariantly match local features of maximum count tiles (top five HS regions of interest (ROIs), designated h1–h5) to the corresponding location of those tiles in WSI tissue area. The uppermost five HSs (based on MF counts) were selected to display a survey of mitotic activity heterogeneity; however, a different number of tiles could be chosen for exhibition. The novel workflow and steps for the automated TMHS are illustrated [Figure 1].
Independent of TMHS algorithm image analysis, the 30 cases were evaluated by pathologists. A pathologist (CHH) examined and annotated digital images using Aperio ImageScope and digital display monitor to demarcate areas of the greatest mitotic activity within specimens. The annotation consisted of a circular ROI corresponding to 2.37 mm2 area (R1) or approximately ten high-power fields of view (FOVs) in standard clinical microscopes with ocular lens eyepiece field number 22 (manuscript in preparation). A pathologist (HAA) counted immunolabeled MF visually and recorded counts electronically (manuscript in preparation). In addition to producing MF counts in the independently designated R1 HS, this pathologist also counted MF in the TMHS-designated h1 HS. The MF count values generated within h1 and R1 HS by the TMHS algorithm and the pathologist, respectively, were compared to counts obtained independently using a second machine algorithm established for cell-object counting algorithm (OCA) (HALO v2.1.1637, Cytonuclear Algorithm v1.4, Indica Labs, Corrales, NM, USA). The OCA algorithm does not include an automated topographic tissue mapping function. The degree to which MF R1 HS identified by the pathologist coregistered with the TMHS algorithm-designated HS (h1–h5) was assessed visually in overlay images.
Further performance appraisal was provided by a group of six pathologists who obtained MF counts as part of a larger concurrent study devoted to comparing pathologist performance agreement between conventional microscopy and digital display of WSI (in preparation). For this comparison, 35 anti-pHH3-immunolabeled canine melanomas were read by pathologists in their individual diagnostic sign-out environments using conventional microscopy. The pathologists independently identified their preferred mitotic HS for evaluation and used the ×40 objective lens for counting MF in ten contiguous nonoverlapping microscopic FOVs. Pair-wise interobserver correlation analyses among pathologists provided contrast with automated TMHS method reproducibility.
MF count values acquired from TMHS, independent pathologist, and OCA were analyzed for methods comparisons by Passing and Bablok linear regression agreement analysis using log10-transformed data. Bland and Altman analysis of MF values derived by the various modalities was performed to gauge the limits of agreement between methods using log10-transformed data. These difference versus average plots included analyses of the following three comparisons:
- Pathologist R1 – TMHS h1 versus (TMHS h1 + pathologist R1)/2
- Pathologist h1 – TMHS h1 versus (TMHS h1 + pathologist h1)/2
- OCA h1 – TMHS h1 versus (TMHS h1 + OCA h1)/2.
Analyses were performed to understand the correlation of MF counts with melanoma patient survival data using Kaplan–Meier survival analysis and the log-rank test. Pair-wise correlation of MF counts among pathologists obtained by conventional microscopy was analyzed using Spearman's (rho) correlation analysis of log10-transformed data. Statistical significance of tests is claimed for P < 0.05.
| Results|| |
Automated quantification and mapping of proliferative activity
The automated TMHS computational process quantified and mapped mitotic activity HS in digital image files of tumor tissue. During the initial steps (Phase I) [Figure 1], image tiles were segregated into groups that either included tissue or tiles that lacked tissue (glass only). Tiles that contained both tissue and glass (tissue edges) were treated as tissue tiles. Each tissue-containing tile was automatically assigned a unique identification number and exported for feature extraction. MF feature extraction employed a combination of color, size, and shape filters to detect MF features corresponding to the red chromogen of pHH3-immunolabeled mitotically active cells developed during IHC [Figure 2]a. This step also compensated for a range of confounding artifacts commonly associated with tissue processing and staining, based on color, size, and shape to permit subtraction of elements such as pigments, dyes, and extraneous objects [Supplemental Figure S2 [Additional file 3]]. Segmentation and extraction filters employed in Phase I were able to identify anti-pHH3-labeled mitotic cells with notable specificity while eliminating background noise due to common artifacts occurring during slide preparation, labeling, and staining. This was confirmed by visually comparing the postprocessed h1 HS tile binary images with the corresponding bright-field micrograph tile images for all cases [Figure 2]. With background subtraction leaving only MF objects in the segmentation stage, image tiles were recorded in separate folders and moved to the second phase [Figure 1].
In Phase II [Figure 1], tiles were converted to grayscale to enable subsequent object binarization, and thresholding filters permitted MF counting as binary objects, per tile area. MF count data were exported to a spreadsheet, and values were rank-ordered according to the highest to lowest tile count [Figure 2]b. Analysis of this series of canine melanoma tumor tissues included a wide range of proliferative activity, from 2 to 793 MF in the h1 HS tile, and assessment of up to 100 tissue-containing image grid tiles generally included the scope of MF counts. A total of 3356 image tiles (mean tissue-containing tiles per specimen = 111.9 [range 29–210]) were processed.
Phase III of the automated TMHS tool function mapped the five tiles with greatest MF counts back to the WSI thumbnail, to plot mitotic activity topographically [Figure 1]. These five tiles, designated HS h1–h5 corresponding to the ordered magnitude of the top five MF ROIs, were matched by tile address back to the RGB color tile tissue folder. Mapping employed SURF, a local tissue rotation invariant feature extraction computer vision algorithm [Figure 2]c. Local features extracted for this function included distinctive structures, objects, and edges occurring in images. Once maximum confidence in the strongest features matching to the WSI for the maximum count tiles was achieved, a bounding box corresponding to the magnitude of count HS tiles h1–h5 was displayed automatically [Figure 2]c and [Figure 3]a.
|Figure 3: Diagnostic decision support graphics provide topography and count metrics for tissue hotspots. Automated mapping of the tissue grid tiles having the most numerous mitotic figure counts across the entire tissue section image is illustrated in the context of the image processing tile grid. Each tile area (2.29 mm2) is commensurate with ten high-power (×40) fields of view on standard clinical microscopes (2.37 mm2 area). (a) Square tiles h1–h5, assigned by the automated topometric mitotic hot spot tool, represent the five tiles with the highest mitotic figure count. In context of the grid regions of interest, the hot spot tiles reveal the topographic heterogeneity of the proliferative activity in this example tumor. The square (red box) depicts the h1 tile with the highest mitotic figure count. (b) Mitotic figure count values are plotted for up to 100 tiles having the greatest counts, providing an indication of the proliferative characteristics across a tissue section. The graphic depicts plots for 30 biopsies. The inset graphic depicts the relative differences in proliferative characteristics within the 50 highest count tiles of 27 tumor specimens. The three most proliferatively active melanoma cases were excluded for illustration purposes|
Click here to view
The automated TMHS algorithm provides diagnostic decision support and tumor grading functions for review by the pathologist. These outputs include graphics of tumor image and overlay tiles depicting the MF HS locations topographically [Figure 3]a; a MF counts slope plot for up to 100 grid tile count values across each tissue image [Figure 3]b, as well as a corresponding spreadsheet of MF count values. Due to the established grid tile metrics, the MF values are scaled based on a standard area in diagnostic surgical pathology equal to ten high-power FOVs, for each tissue-containing tile [Supplemental Table 1 [Additional file 4]]. The automated TMHS diagnostic support algorithm function was executed through a GUI controller [Supplemental Figure S1 [Additional file 5]].
Automated topometric hotspot performance validation
Automated TMHS functionality was validated further in a representative clinical application using the series of 30 canine melanoma biopsies immunolabeled with anti-pHH3 IHC for proliferating cells. Tumor mitotic activity was ascertained, and counts were obtained and mapped. Several measures were compared for evidence of correlation, reproducibility, and method interchangeability. For these comparisons, the automated TMHS machine performance output was correlated with the pathologists who independently identified, mapped, and counted tumor mitotic HS (R1) in the case series. Additional appraisal of the TMHS performance was made by comparing the automated TMHS-generated h1 MF counts with counts made visually by the pathologist from the identical TMHS-designated h1 HS. A third comparison was made between the TMHS algorithm count values and those obtained by quantifying the immunolabeled MF within the TMHS-designated h1 HS using a second machine OCA. This OCA was independently established using the same immunolabeled MF objects of interest in the WSI; however, the OCA algorithm does not include a mapping function. In addition, mitotic counts generated by the pathologist and from the TMHS algorithm were tested for prognostic utility by attempting to stratify patient survival intervals.
Mitotic counts are a prognostic indicator of survival/prognosis in canine melanoma, similar to the clinical utility of determining MF frequency in several human tumor types, including melanoma.,,,,, Therefore, an examination predicting longer versus shorter survival time risk, based on MF values from HS, was undertaken analogous to how the automated TMHS tool would be applied clinically. A MF cut score value of 110 immunolabeled MF was determined for dogs in this study through a process of serially testing candidate MF count values to segregate patient populations based on survival postdiagnosis. Kaplan–Meier plot and log-rank analysis demonstrate a difference in survival, with a cut point of 110 pHH3-positive cells [[Supplemental Figure S3 [Additional file 6]], for TMHS, P = 0.0034; and for manual evaluation, P = 0.0013]. The median survival of the poor prognostic group was 45 or 38 days and for the more favorable prognostic group, 197.5 or 197 days, based on TMHS and pathologist, respectively.
The extent of topographic agreement between the automated TMHS algorithm-mapped HSs (h1–h5) and the R1 HS manually identified by the pathologist was evaluated using image overlays of these two independent approaches to HS identification [Supplemental Figure S4 [Additional file 13]]. Topographically, the manual primary HS (R1) partially or more substantially coincided with one or more of the automated mitotic HS (h1–h5) in the majority of cases (23 of 30 [76.7%]) [Supplemental Table 1 [Additional file 7]]. In one of the seven cases that lacked co-registration, the survival prognosis based on mitotic activity was misjudged by the pathologist. Another discrepancy revealed a distinct HS focus that the pathologist identified, which shared a MF count similar to the TMHS-designated h5 HS. This latter discrepancy may not be clinically meaningful, whereas the former one in which the mitotic HS is not aligned with the representative tumor mitotic activity represents a variance that risks leading to diagnostic error.
The agreement among MF count values obtained by the automated TMHS mapping algorithm, OCA machine analyses, and by pathologist visual inspection was also considered through nonparametric Passing and Bablok linear regression for method comparison. Examining MF count correlation in this manner allows for measurement error (imprecision) in the approaches compared, does not require the measurement error to be normally distributed, and is insensitive to outliers. This analysis provided strong evidence of agreement among the various approaches [Figure 4]a,[Figure 4]b and [Figure 4]c, P < 0.0001]. The degree of correlation was greater when precisely the same area was assessed (h1 compared to h1, in contrast to R1 compared to h1), with stronger parallel observed between OCA and TMHS in most of the cases, compared to visual counts of MF by the pathologist. Two data points deviating most from the regression line were noted [Figure 4]c. Both cases were influenced by a varying and regionally weak IHC chromogenic signal development [lower point, ID 3043, upper point, ID 4583, [Supplemental Table 1 [Additional file 8]]. Both algorithms were impacted, resulting in undercounts, while the pathologist appears to have compensated in these instances [Supplemental Table 1 [Additional file 9]].
|Figure 4: Performance fit comparisons between the automated topometric mitotic figure hotspot tool (TMHS) and mitotic figure quantification by expert visual assessment or a second machine object counting algorithm (OCA). Regression scatter plots of paired mitotic figure count observations (log10) from the TMHS h1 hot spot, with (a) counts independently obtained by pathologist (R1 hotspot), (b) counts of the automated h1-selected topometric hotspot obtained by the pathologist, or (c) counts from the automated h1-selected topometric hotspot obtained by object counting algorithm counting. Two data points deviating from the regression line were noted in particular. One of these cases displayed in c was influenced by a weaker immunohistochemistry signal development (lower point, ID 3043). In the second case (upper point, ID 4583), repeat counts obtained by the pathologist ranged from 14 to 16 for both h1 and R1 hotspots, which were regions of interest completely overlapping topographically. Blue-shaded areas represent 95% confidence intervals. R2 values and regression equations are indicated on the figure. All comparisons represented significant correlations|
Click here to view
Similarly, paired assessments of MF HS were depicted in a series of difference versus average plots [Figure 5]. Bland and Altman analysis estimates the degree to which two methods differ in quantitative measurement (limits of agreement) and can thereby aid in deciding if one method can be substituted for another. The limits of agreement analyses indicate that methods were interchangeable [Figure 5].
|Figure 5: Limits of agreement. The difference in measurements made in paired analyses was plotted against the average of two methods compared (all data log10transformed). General agreement between methods is evident with minimal variance (limits of agreement, shaded area). Bland–Altman scatter plots of paired count observations (a) obtained by pathologist-independent visual assessment of R1 minus automated h1-selected topometric hotspot, (b) pathologist counts by visual inspection from the h1-selected topometric hotspot minus counts generated from the automated h1-selected topometric hotspot, or (c) mitotic figure counts obtained from the h1-selected hotspot using a second machine object counting algorithm (OCA) minus the automated h1-selected topometric hotspot counts|
Click here to view
The reproducibility of the automated TMHS algorithm HS mapping and counting functions was further assessed on the identical image files following 90° in-plane image rotation, to create an alternative input orientation. The resulting h1 HS topographic location and the respective MF counts substantially reproduced the initial output when the tissue image was reprocessed by the automated TMHS algorithm in a different orientation [Supplemental Table 1 [Additional file 10]] and [Figure 6]a, [Figure 6]b. The automated TMHS h1 HS was registered to within a few hundred microns of its original tissue location in all but five specimens [Supplemental Table 1 [Additional file 11]]. In four of the five cases, the new h1 HS in the rotated image overlapped or was adjacent to the original h2 HS, the ROI with the second highest mitotic count. In two of these four cases for which the newly mapped h1 HS was aligned with the original h2 HS after image rotation, the counts were within 3 MF of the initial h1 mitotic count values and not >9 MF among the remaining two [Supplemental Table 1 [Additional file 12]]. Repeating the automated TMHS MF count values using images in a second orientation provided evidence of excellent correlation for the mapping and counting functions (R2 = 0.9916) [Figure 6]c. This degree of reproducibility contrasts with a range of relatively less well-correlated MF values evident when assessing MF agreement among a group of pathologists [Figure 6]d. The latter data were obtained evaluating anti-pHH3-labeled mitotic activity in a series of melanomas by microscopy, under typical case sign-out circumstances. This operational discordance when pathologists perform this task remains a concern, and the functional reproducibility of the automated TMHS method is a strong characteristic of the tool and represents a distinct advantage.
|Figure 6: Performance fit comparisons between repeated automated TMHS analyses and among a group of pathologists examining anti-phosphorylated histone H3-immunolabeled melanomas by conventional microscopy. (a) For TMHS, an original input image, with grid tile overlay, includes the computed TMHS h1 hotspot indicated by red box tile, which is illustrated in relation to the R1 hotspot selected by the pathologist (circle). (b) Identical image as in a, after 90° in-plane rotation, is processed in TMHS to demonstrate the computed h1 hotspot within the same tissue region as the original. The R1 hotspot, selected by the pathologist and shown as an image overlay, has the same tissue coordinates as in a. Bar = 1.514 mm. (c) Regression scatter plot of paired mitotic figure count values in the two image orientations among the 30 cases, from automated TMHS mapping tool (log10transformed). Blue-shaded area represents 95% confidence interval. R2 value and regression equation are indicated on the figure. Measurements from the two image orientations were highly significantly correlated (P < 0.0001). (d) Pair-wise, interobserver agreement of mitotic figure assessment by each pathologist using conventional microscopy on anti-phosphorylated histone H3-immunolabeled melanomas (n = 35). Each cell in the heat map represents the Spearman correlation coefficient (rho values, bar graph) between each pair of pathologists (A, B, F, J, K, and L), using the total mitotic figure count in ten high-power microscope fields of view|
Click here to view
| Discussion|| |
Assessment of proliferative capacity is an important component of tumor grading and treatment planning for several types of malignancies, including breast, adrenal gland, meningiomas, neuroendocrine carcinomas, soft-tissue sarcomas, and melanoma.,,,,,,,, Yet, during routine visual inspection of H&E-stained biopsies, pathologists frequently fail to agree on which structures represent MF. In a recent study, only 21 of 92 MF in H&E-stained tumor biopsies were unanimously identified as MF by all five participating pathologists examining the same 40 FOVs (https://nciphub.org/groups/eedapstudies/wiki/Presentation: AReaderStudyona14headMicroscope December 3, 2018). Selection bias, sample size, and tumor heterogeneity can all contribute to disparities. The use of immunohistochemical methods to label cell proliferation markers has been shown to improve MF detection and quantitation for prognostic purposes.,,,, The approach demonstrated here takes advantage of the differential contrast and target probe enhancement of mitotic (M) phase cells, through pHH3 immunolabeling. These characteristics are commonly shared among tumor types and even across species. While prognostic grading can be accomplished using MF counts from H&E-stained tissues and mitotic immunolabeled tissues, the absolute values for the prognostic cut points differ between cancer types and mitotic counts obtained by evaluation of H&E stains and pHH3 IHC.
Furthermore, quantifying MF from tumor HS using automated computer assistance stands to improve the interobserver variability inherent in clinical MF assessment. The automated TMHS algorithm uses an unsupervised tile-based ROI approach, executed through a GUI, fully processing each image pixel in subdivided static images at approximately ×4 magnification (size ≈ KB). Computationally, this helps to simplify processing requirements manifest in some other systems. For example, the automated selection of hotspot (ASH) tool, applied in the detection of HS in melanomas and adrenocortical tumors, differs from TMHS by the block size, and ASH image tiles number in the thousands (size ≈ GB), which together would appear to add substantially to processing requirements. By contrast, the automated TMHS approach permitted HS detection from the first 100 tissue tiles. TMHS analysis indicated that MFs were concentrated within ≤80–100 tiles, approximating a tissue surface area of ≤240 mm2. In addition, TMHS operates with lower resolution input images from ×20 magnification optical scans, which can be more rapidly acquired, providing further advantageous image processing distinctions from other methods.,, Consequently, these TMHS design characteristics afford relative computational efficiency for handling the large size of WSI data files.
Clinical practice typically includes assessment of MF density for areas of tumor mitotic HS. Customarily, counts are made beginning in an area of the tumor with the most plentiful MF, i.e., the mitotic HS. Many MF cut points used for tumor grading are based on this approach.,, Prognostic assessment of proliferative activity by microscopy usually involves examination of ten high-power magnification FOVs (×40 objective lens).,,,, Significant differences in the predictive value of MF counts assessed from mitotic HS, in contrast to MF counts obtained from either randomly selected FOV, or the tumor area more globally, have been documented., While counts obtained from mitotic HS are clearly the preferred practice, the visual determination of HS location is subjective, time-consuming, error prone, and thereby influences the prognostic values represented., In general, results obtained can be poorly reproducible, which can be attributed to factors such as tissue areas chosen, structures identified, and counts acquired by different pathologists. In our study, TMHS-designated HS locations co-registered topographically with mitotic HS manually selected by the pathologist, overlapping in approximately 77% of cases. This level of intermodality agreement between pathologist and TMHS machine was principally reinforced by the analysis of survival prognosis of the 30 dogs with melanoma. MF count values produced with the automated TMHS method, and independently by the pathologist, both resulted in the ability to predict patient survival differences, although intermodal discrepancy resulted in different prognostic grading for one case. By contrast, analysis of cutaneous melanoma revealed that pathologist manual versus automated HS topographies were less commonly co-registered (67%) than for the automated TMHS analysis (77%), even though skin biopsy tissue areas (range: 0.020–50 mm2) were significantly smaller than mucosal melanomas overall. Furthermore, in cutaneous melanoma, the clinical utility of the automated HS-acquired pHH3-immunolabeled MF counts was judged prognostically inferior to the manually derived values, a less optimal distinction compared to the TMHS algorithm performance shown here. In part, this is likely due to improvements in MF object segmentation and feature extraction made using the automated TMHS tool, contributing therefore to enhanced precision.
Low MF counts can also be prognostically informative and the automated TMHS global analysis approach provides such insight when limited proliferative capacity is manifest in biopsies. The quantifying and mapping features of some MF detection systems appear to be more constrained than TMHS. For example, testing of all tissue locations is not always completely inclusive, and sparse small clusters of MF may be excluded, or counts are not yielded in each case, as further improvements in nuclear segmentation are required.
The image analysis process must reproducibly distinguish mitotic cells from a variety of potentially confounding structures and artifacts in tissues. Addressing tissue artifacts, while assuring the quality of MF recognition, segmentation, and counting, remain key challenges. The automated TMHS tool confronts these issues, and essential functional reproducibility is demonstrated. Feature extraction and quantitation of pHH3-immunolabeled MF using automated TMHS were highly correlated with a second independently established OCA although it is noteworthy that the OCA does not map HS localization topography. Likewise, automated TMHS function was significantly correlated with the pathologist, an outcome requiring both MF segmentation and HS mapping precision. Together, these findings provide substantial confidence for the nuclear segmentation function and the avoidance of noise in the form of artifacts. Additional assurance was provided by qualitative review of the MF feature segmentation markup by the pathologist for all algorithms on all specimens, using the computer display. This important decision support feature is not a universal element in other HS detection methods.
Tissue- and stain-related artifacts inherent in biopsy tissue sections must be addressed for proper performance. We encountered a range of artifacts that could not be filtered out by selecting a single approach; therefore, a series of filters were designed to deal with a number of common artifacts. TMHS development addressed typical artifacts hampering previous algorithms primarily incorporating commercially available CAD or other open source tools., For example, TMHS was able to circumvent melanin, hemosiderin, and other pigments, in contrast to previous tools., Red chromogenic signal development likely contributed to this. In addition, automated TMHS processing otherwise eliminated a need for supplemental postprocessing required to address false-positive HS due to sporadic off-target labeling, and the necessity to exclude tissue folds and necrosis by employing independent preprocessing steps was averted., In distinction to other systems,, particular tissue artifacts that necessitated the exclusion of 8%–20% of cases altogether were not similarly problematic for TMHS mapping. Previously, artifact complications were reduced using a combination of Unser features and local binary pattern. However, the tools were combined in a series of support vector machine applications of Gaussian kernel function that required machine learning. By contrast, automated TMHS proliferation quantification and HS mapping programming is a feature-based algorithm that does not comprise machine learning.
A feature-based approach requires specimens preserved and processed consistently. Tissue processing factors have the potential to impact the success of the automated TMHS algorithm and remain perennial issues across all types of tissue image analysis. In the present study, there were two cases for which the analyses appeared to deviate and for which the level of agreement was less representative of the remaining samples [Figure 4]c. Anti-pHH3-immunolabeling signal appeared to be faint in some tissue areas in these two cases, which was apparent on the quality assurance review process. However, this circumstance resulted in undercount of some MFs, which were excluded by TMHS algorithm threshold settings established to subtract background off-target noise. The efficiency of the automated TMHS system rests in part on proper probe recognition of MF and its signal generation. The clinical implementation of automated TMHS will necessitate adherence to standardized preanalytic control procedures to maximize specimen integrity in preparation for tissue processing, image acquisition, and computation. Successfully accomplishing unsupervised image processing for MF HS detection is reliant upon uniform reproducible tissue labeling and signal contrast generation for efficient computer vision function.
| Conclusions|| |
HS detection and mitotic counting plays a crucial role in proliferation assessment and tumor grading.,,,,,,,,, In this paper, we established a method of HS location and mapping that does not rely upon machine learning but by contrast is tile-based and computationally less complex in comparison to other methods which utilize algorithm processing power across an entire WSI. Identification of tissue HS, assessments of mitotic activity, and the utility of the automated TMHS functions were all shown to be clinically relevant, prognostically predictive, and highly correlated with the pathologist tasked to directly challenge the machine. Limits of agreement correlation analyses indicated that the automated TMHS method was clinically interchangeable with the pathologist using digital display of WSI. Technical quality of MF immunolabeling and IHC chromogenic signal development could be a factor in our automated computational approaches, whereas pathologist compensation for varying and regionally weak IHC was apparent in at least two cases. Notwithstanding, the overarching goal of providing improved clinically actionable prognostic decision support free from the widely recognized interobserver variability prone to traditional MF assessments,,,,,, was achieved. The reproducibility of the automated TMHS algorithm proved superior compared to replicates among six pathologists using routine microscopy, which included identifying their own mitotic HSs as in a clinical practice setting. Utilizing the reproducible precision of the computer-assisted TMHS diagnostic mapping function will enhance pathologist tumor prognostic grading efficiency across a range of malignancies with varying clinical MF count cut points.
Specimens and patient survival data were provided as a consequence of collaborations with Drs. Michael Goldschmidt, University of Pennsylvania, and EJ Ehrhart, Colorado State University in addition to those from co-authors. Brandon D. Gallas, PhD, Weijie Chen, PhD, and Qi Gong, MS, from the US Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging, Diagnostics, and Software Reliability provided input into the study design and analysis concepts of the mitotic figure imaging studies with the pathologists, for which we are grateful. Current addresses for relocated authors include CHH, Pfizer Drug Safety Research and Development, 455 Eastern Point Rd., Groton, CT 06340; SDC, Tox Path Specialists, LLC., 8420 Gas House Pike, Suite G, Frederick, MD 21701; LTL, Department of Comparative Pathobiology, Purdue University College of Veterinary Medicine, 725 Harrison Street, West Lafayette, IN 47907; MDS, Antech Diagnostics, 410 Union Ave., Framingham MA, 01702; RPT, Antech Diagnostics, Lacey, WA, and the Comparative Pathology Branch, US Army Medical Research Institute of Chemical Defense, Aberdeen Proving Ground, MD.
Financial support and sponsorship
This research was supported by the Intramural Research Program, Center for Cancer Research, National Cancer Institute, Bethesda, Maryland. This work utilized the computational resources of the NIH High Performance Computing Biowulf cluster (http://hpc.nih.gov). Commercial and academic licensing inquiries can be addressed to Dr. Eric Chen, NCI Technology Transfer Center, firstname.lastname@example.org; NIH reference E-038-2019-0.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Al-Janabi S, van Slooten HJ, Visser M, van der Ploeg T, van Diest PJ, Jiwa M. Evaluation of mitotic activity index in breast cancer using whole slide digital images. PLoS One 2013;8:e82576.
Kim YJ, Ketter R, Steudel WI, Feiden W. Prognostic significance of the mitotic index using the mitosis marker anti-phosphohistone H3 in meningiomas. Am J Clin Pathol 2007;128:118-25.
Shen S, Wolfe R, McLean CA, Haskett M, Kelly JW. Characteristics and associations of high-mitotic-rate melanoma. JAMA Dermatol 2014;150:1048-55.
Azzola MF, Shaw HM, Thompson JF, Soong SJ, Scolyer RA, Watson GF, et al.
Tumor mitotic rate is a more powerful prognostic indicator than ulceration in patients with primary cutaneous melanoma: An analysis of 3661 patients from a single center. Cancer 2003;97:1488-98.
Coindre JM. Grading of soft tissue sarcomas: Review and update. Arch Pathol Lab Med 2006;130:1448-53.
Rakha EA, Reis-Filho JS, Baehner F, Dabbs DJ, Decker T, Eusebi V, et al.
Breast cancer prognostic classification in the molecular era: The role of histological grade. Breast Cancer Res 2010;12:207.
Voss SM, Riley MP, Lokhandwala PM, Wang M, Yang Z. Mitotic count by phosphohistone H3 immunohistochemical staining predicts survival and improves interobserver reproducibility in well-differentiated neuroendocrine tumors of the pancreas. Am J Surg Pathol 2015;39:13-24.
Donizy P, Kaczorowski M, Leskiewicz M, Zietek M, Pieniazek M, Kozyra C, et al.
Mitotic rate is a more reliable unfavorable prognosticator than ulceration for early cutaneous melanoma: A 5-year survival analysis. Oncol Rep 2014;32:2735-43.
Francken AB, Shaw HM, Thompson JF, Soong SJ, Accortt NA, Azzola MF, et al.
The prognostic importance of tumor mitotic rate confirmed in 1317 patients with primary cutaneous melanoma and long follow-up. Ann Surg Oncol 2004;11:426-33.
Olar A, Wani KM, Sulman EP, Mansouri A, Zadeh G, Wilson CD, et al.
Mitotic index is an independent predictor of recurrence-free survival in meningioma. Brain Pathol 2015;25:266-75.
Veta M, van Diest PJ, Jiwa M, Al-Janabi S, Pluim JP. Mitosis counting in breast cancer: Object-level interobserver agreement and comparison to an automatic method. PLoS One 2016;11:e0161286.
Bonert M, Tate AJ. Mitotic counts in breast cancer should be standardized with a uniform sample area. Biomed Eng Online 2017;16:28.
Dessauvagie BF, Thomas C, Robinson C, Frost FA, Harvey J, Sterrett GF. Validation of mitosis counting by automated phosphohistone H3 (PHH3) digital image analysis in a breast carcinoma tissue microarray. Pathology 2015;47:329-34.
Khan Niazi MK, Yearsley MM, Zhou X, Frankel WL, Gurcan MN. Perceptual clustering for automatic hotspot detection from ki-67-stained neuroendocrine tumour images. J Microsc 2014;256:213-25.
Nielsen PS, Riber-Hansen R, Schmidt H, Steiniche T. Automated quantification of proliferation with automated hot-spot selection in phosphohistone H3/MART1 dual-stained stage I/II melanoma. Diagn Pathol 2016;11:35.
Rezanko T, Akkalp AK, Tunakan M, Sari AA. MIB-1 counting methods in meningiomas and agreement among pathologists. Anal Quant Cytol Histol 2008;30:47-52.
Winther TL, Arnli MB, Salvesen Ø, Torp SH. Phosphohistone-H3 proliferation index is superior to mitotic index and MIB-1 expression as a predictor of recurrence in human meningiomas. Am J Clin Pathol 2016;146:510-20.
Swiderska-Chadaj Z, Markiewicz T, Grala B, Lorent M. Content-based analysis of ki-67 stained meningioma specimens for automatic hot-spot selection. Diagn Pathol 2016;11:93.
Lu H, Papathomas TG, van Zessen D, Palli I, de Krijger RR, van der Spek PJ, et al.
Automated selection of hotspots (ASH): Enhanced automated segmentation and adaptive step finding for ki67 hotspot detection in adrenal cortical cancer. Diagn Pathol 2014;9:216.
Lopez XM, Debeir O, Maris C, Rorive S, Roland I, Saerens M, et al.
Clustering methods applied in the detection of ki67 hot-spots in whole tumor slide images: An efficient way to characterize heterogeneous tissue-based biomarkers. Cytometry A 2012;81:765-75.
Huang CH, Veillard A, Roux L, Loménie N, Racoceanu D. Time-efficient sparse analysis of histopathological whole slide images. Comput Med Imaging Graph 2011;35:579-91.
Simpson RM, Bastian BC, Michael HT, Webster JD, Prasad ML, Conway CM, et al.
Sporadic naturally occurring melanoma in dogs as a preclinical model for human melanoma. Pigment Cell Melanoma Res 2014;27:37-47.
Gillard M, Cadieu E, De Brito C, Abadie J, Vergier B, Devauchelle P, et al.
Naturally occurring melanomas in dogs as models for non-UV pathways of human melanomas. Pigment Cell Melanoma Res 2014;27:90-102.
Hernandez B, Adissu HA, Wei BR, Michael HT, Merlino G, Simpson RM, et al.
Naturally occurring canine melanoma as a predictive comparative oncology model for human mucosal and other triple wild-type melanomas. Int J Mol Sci 2018;19. pii: E394.
Otsu N. Threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 1979;9:62-6.
Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded-up robust features (SURF). Comput Vis Image Underst 2008;110:346-59.
Passing H, Bablok W. A new biometrical procedure for testing the equality of measurements from two different analytical methods. Application of linear regression procedures for method comparison studies in clinical chemistry, part I. J Clin Chem Clin Biochem 1983;21:709-20.
Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8:135-60.
Bergin IL, Smedley RC, Esplin DG, Spangler WL, Kiupel M. Prognostic evaluation of ki67 threshold value in canine oral melanoma. Vet Pathol 2011;48:41-53.
Papathomas TG, Pucci E, Giordano TJ, Lu H, Duregon E, Volante M, et al.
An international ki67 reproducibility study in adrenal cortical carcinoma. Am J Surg Pathol 2016;40:569-76.
Tsuta K, Liu DC, Kalhor N, Wistuba II, Moran CA. Using the mitosis-specific marker anti-phosphohistone H3 to assess mitosis in pulmonary neuroendocrine carcinomas. Am J Clin Pathol 2011;136:252-9.
Villani V, Mahadevan KK, Ligorio M, Fernández-Del Castillo C, Ting DT, Sabbatino F, et al.
Phosphorylated histone H3 (PHH3) is a superior proliferation marker for prognosis of pancreatic neuroendocrine tumors. Ann Surg Oncol 2016;23:609-17.
Nielsen PS, Riber-Hansen R, Jensen TO, Schmidt H, Steiniche T. Proliferation indices of phosphohistone H3 and ki67: Strong prognostic markers in a consecutive cohort with stage I/II melanoma. Mod Pathol 2013;26:404-13.
Yi F, Huang J, Yang L, Xie Y, Xiao G. Automatic extraction of cell nuclei from H&E-stained histopathological images. J Med Imaging (Bellingham) 2017;4:027502.
Kothari S, Phan JH, Stokes TH, Wang MD. Pathology imaging informatics for quantitative analysis of whole-slide images. J Am Med Inform Assoc 2013;20:1099-108.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6]