|J Pathol Inform 2018,
Twenty years of digital pathology: An overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives
Liron Pantanowitz1, Ashish Sharma2, Alexis B Carter3, Tahsin Kurc4, Alan Sussman5, Joel Saltz4
1 Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
2 Department of Biomedical Informatics, Emory University, GA, USA
3 Department of Pathology and Laboratory Medicine, Children's Healthcare of Atlanta, GA, USA
4 Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
5 Department of Computer Science, University of Maryland, College Park, MD, USA
|Date of Submission||11-Sep-2018|
|Date of Acceptance||28-Oct-2018|
|Date of Web Publication||21-Nov-2018|
Dr. Liron Pantanowitz
Department of Pathology, University of Pittsburgh Medical Center Shadyside, Cancer Pavilion Suite 201, 5150 Centre Avenue, Pittsburgh, PA 15232
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Almost 20 years have passed since the commercial introduction of whole-slide imaging (WSI) scanners. During this time, the creation of various WSI devices with the ability to digitize an entire glass slide has transformed the field of pathology. Parallel advances in computational technology and storage have permitted rapid processing of large-scale WSI datasets. This article provides an overview of important past and present efforts related to WSI. An account of how the virtual microscope evolved from the need to visualize and manage satellite data for earth science applications is provided. The article also discusses important milestones beginning from the first WSI scanner designed by Bacus to the Food and Drug Administration approval of the first digital pathology system for primary diagnosis in surgical pathology. As pathology laboratories commit to going fully digitalize, the need has emerged to include WSIs into an enterprise-level vendor-neutral archive (VNA). The different types of VNAs available are reviewed as well as how best to implement them and how pathology can benefit from participating in this effort. Differences between traditional image algorithms that extract pixel-, object-, and semantic-level features versus deep learning methods are highlighted. The need for large-scale data management, analysis, and visualization in computational pathology is also addressed.
Keywords: Computational pathology, digital pathology, image analysis, informatics, vendor neutral archive, whole-slide image
|How to cite this article:|
Pantanowitz L, Sharma A, Carter AB, Kurc T, Sussman A, Saltz J. Twenty years of digital pathology: An overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J Pathol Inform 2018;9:40
|How to cite this URL:|
Pantanowitz L, Sharma A, Carter AB, Kurc T, Sussman A, Saltz J. Twenty years of digital pathology: An overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J Pathol Inform [serial online] 2018 [cited 2019 Feb 21];9:40. Available from: http://www.jpathinformatics.org/text.asp?2018/9/1/40/245903
| Introduction|| |
Since the commercial introduction of whole-slide imaging (WSI), digital pathology has developed into a thriving field., The ability to digitize an entire pathology glass slide has been transformational and engendered numerous clinical, educational, and research applications. The capacity to rapidly generate large quantities of microscopy information has become feasible as a result of ongoing technical advances in optical imaging technologies. The first digital microscope systems cost about $300,000 to set up and took over 24 h to scan a single slide. These devices have advanced significantly over the past 20 years. Today, whole-slide scanners are capable of automatically producing very high-resolution images that replicate glass slides (so-called virtual microscopy). Equally dramatic improvements in storage and computational technology have made possible the processing of large WSI datasets. As a result, WSI devices have become important tools that can support routine diagnostic work and scientific discovery in pathology and have enabled the development of next-generation tools including artificial intelligence (AI).
This article provides an overview of WSI and the state-of-the-art, promise and challenges of digital pathology. It presents a compendium of past and present efforts to develop devices and infrastructure to scan, catalog, and store large collections of WSIs. This article also reviews the methods and tools used to carry out large-scale, integrated analyses with visualization, interrogation, and mining of WSI for research and clinical decision-making. Integration is the key to correlate WSI data with clinical, radiological, and genomic information. A number of examples are also described from different institutions and consortia that launched into very large-scale digital pathology data acquisition, storage, and integration efforts.
| Evolution of the Virtual Microscope|| |
Current software systems and methods for WSI data management, query and viewer methods arose from the 1990s era of computer science spatial dataset research. This research was initially motivated by the need to analyze, visualize, and query sensor data acquired from satellites and from basic and applied earth science applications. Through the use of this software, raw data acquired by orbiting satellites could be associated with appropriate geospatial regions, spatially related data could be aggregated, and useful datasets could be produced.
In the context of a National Science Foundation Grand Challenge Award, the Saltz group developed software – the Active Data Repository (ADR) system – designed to carry out combined retrieval and processing of large amounts of spatio-temporal data. This software motivated the development of innovative spatial indexing methods and also made use of the map-reduce paradigm later employed by Hadoop and Spark. Motivated by a discussion with Sal Pizzo, the pathology chair at Duke concerning eventual replacement of microscopes with computers in anatomic pathology, Saltz realized that the methods employed for satellite and earth science data could also be applied to the very large datasets obtained by digitization of WSIs. The realization that this nascent technology, being developed to process satellite data, could be applied to WSIs was the nidus that triggered the development of software necessary to support the earliest Virtual Microscope [Figure 1] that was developed from 1996 to 1998.
|Figure 1: Screenshot of the early Johns Hopkins/University of Maryland “Virtual Microscope” whole-slide imaging viewer client|
Click here to view
When the Saltz group launched their “Virtual Microscope” research in 1996, there were no whole-slide scanners, so data were acquired by tiling WSIs using a platform designed to acquire photomicrographs. The initial Virtual Microscope software used the ADR to carry out spatially accessed data retrieval and to generate output data at varying levels of magnification. This initial prototype system was soon refined to support data caching, prefetching, support for simultaneous queries from multiple clients, and precomputed image pyramids. A follow-on system employed a different backend architecture called DataCutter. Both the ADR and DataCutter were also adapted to support visualization [Figure 2] and [Figure 3] as well as analysis and visualization of three-dimensional (3D) images obtained from serial sections.,
|Figure 2: Architecture of the active data repository spatial data query system used as the virtual microscope backend|
Click here to view
|Figure 3: Architecture of the DataCutter system used to support the Virtual Microscope application|
Click here to view
Flexible, high-end software systems such as the ADR, DataCutter, Hadoop geographic information system (GIS), and SPARK GIS have continued to be used for applications such as rendering 3D pathology microanatomic objects and complex problems in pathology spatial analytics, but over the ensuing years, a variety of software systems have been developed to support multiresolution two-dimensional (2D) dataset traversal. It was recognized that in many cases, all that was required to be supported was to pan and zoom across and into multi-resolution 2D datasets represented by image pyramids. Zoomify, Keyhole EarthViewer (acquired by Google and integrated into Google Earth), Lizardtech (now Extensis) GeoExpress, and Microsoft's Seadragon technology were pioneering examples of systems of this kind. These systems have continued to evolve. There is now an open-source variant of Seadragon (Open Seadragon).
Today, a variety of companies produce commercial WSI scanners, and these systems acquire WSIs in proprietary formats, either via tile-based or line-based scanning. The capability to transform data between some of these formats is provided by the OpenSlide library, a vendor neutral C library. A standard whole-slide format developed by the Digital Imaging and Communications in Medicine (DICOM) standards committee working group (WG) 26 (Supplement 145) has defined standards for tiled pyramid-based data storage and frame of reference. There are currently a variety of actively supported open-source WSI systems, in addition to proprietary whole-slide software systems supported by vendors. Examples of such systems are caMicroscope, the Digital Slide Archive, the Sedeen viewer, and QuPath.,,, These Virtual Microscope software systems make use of image pyramid-based data management systems to support multiresolution pan-and-zoom operations. Each of these systems supports invocation of image analysis software, and to varying degrees, each also supports management, query, and visualization of imaging features obtained by image analysis algorithms.
| Whole-Slide Imaging Milestones|| |
WSI (previously referred to as virtual microscopy) involves scanning (digitization) glass slides to produce digital slides (also known as WSIs or eSlides). Such digitization is performed on a WSI scanner. The first commercial slide scanner, called the BLISS (Bacus Laboratories Inc., Slide Scanner) system (http://www.jamesbacus.com/page10.html), was designed by James Bacus in 1994. Bacus Research Laboratory were awarded the first two WSI patents. These were filed in 1997 and 1998, so this industrial work was contemporaneous with the above-described academic efforts. Bacus was later acquired by Olympus. InterScope Technologies was founded in 1998 by Dr. Michael Becich et al. associated with the University of Pittsburgh Medical Center. InterScope's ultrafast robotic whole-slide digitizer supported integrated slide imaging, case flow, and clinical data management applications. InterScope was acquired by Trestle Holdings in 2005. Trestle developed a robotic telepathology system (MedMicroscopy) that supported dynamic image capture along with motorized panning and zooming. Tresle was acquired by Chromavision (later renamed Clarient), and their first product was an automated cellular imaging system (ACIS). The ACIS system was capable of quantitative analysis, rare event detection, object counting, and incorporated tissue microarray technology and an interface to Dako automated stainers. Clarient split into a telepathology business unit later bought by Carl Zeiss, and a companion diagnostic company with Dako. In 2000, Aperio developed a linear array detector synchronized with a positioning stage to acquire and compose image strips obtained from successive scans of each specimen. In 2004, Dr. Ronald Weinstein developed a highly innovative scanning technology consisting of a microscope array designed to rapidly acquire WSI data; he created a company called DMetrix to commercialize the technology. Aperio also developed a multi-sensor array approach where a microscope objective lens projects an image on to multiple sensors, allowing concurrent image acquisition and data processing. Today, many novel scanners have been built incorporating state-of-the-art optics, robotics, and computers. These instruments typically contain a microscope with one or more objective lenses, digital cameras, robotics, and numerous other parts (e.g., slide trays/cartridges, light source, barcode reader). Currently, there are a plethora of WSI scanners on the market that cater to a variety of uses.
A contemporary WSI system includes two integrated components [Figure 4]: (1) the scanner that handles image acquisition and (2) the workstation that includes a monitor. According to published guidelines by the College of American Pathologists for clinical validation of WSI, it is unnecessary to separately validate each component of a WSI system. Over the years, manufacturers have continually developed scanners to address clinical needs. Some of these innovations include devices that offer brightfield and/or fluorescence light sources, oil magnification, Z-stacking, and slide capacities that range from holding 1 to 1000 slides. Z-stacking capability is especially valuable for laboratories that scan cytology specimens. Several newer scanners [Figure 5] offer hybrid WSI scanning and a live view mode that permits glass slides to be viewed in real time by remotely controlling magnification, navigation, and focus function with robotics. Such hybrid scanners are particularly desirable for intraoperative consultations accomplished by means of telepathology.
|Figure 4: Overview of a whole-slide imaging system comprised of integrated scanner and workstation components (Image reproduced with permission)|
Click here to view
|Figure 5: Various hybrid whole slide imaging/live robotic instruments showing the number and magnification of objectives for each device|
Click here to view
There are many clinical and nonclinical uses of WSI. Of all the potential use cases, the one that has caused the most excitement and anxiety is the utilization of WSIs for making primary diagnoses. Numerous validation studies have been published indicating that WSI is a reliable tool for routine diagnosis in surgical pathology.,,, For most subspecialties, the published data show excellent correlation between diagnoses made with WSI and conventional light microscopy [Table 1]. A meta-analysis of publications regarding WSI compared with light microscopy revealed 355 out of 8069 discordant instances (4%), of which 109 cases (32%) were related to diagnosis or grading of dysplasia and 32 cases (10%) were ascribed to the inability to find a small object. Cytology no longer appears to be an exception as similar positive findings have been recently demonstrated using WSI for cytopathology. Not surprisingly, several pathology labs around the world have successfully gone “fully digital.”,, Indeed, WSI technology has many advantages over conventional microscopy such as portability, ease of sharing, and retrieval of images, workload balancing, and image analysis.
|Table 1: Published results of whole-slide imaging validation studies for primary diagnosis in different subspecialties|
Click here to view
Despite the aforementioned accomplishments, the adoption of WSI in clinical practice has been relatively slow. This has been attributed to technical, cultural, financial, and, in the United States, regulatory barriers. [Table 2] highlights several milestones related to regulations in the United States. Major progress was made when the Food and Drug Administration (FDA) allowed marketing the first WSI system for primary diagnosis in surgical pathology. Following a pivotal noninferiority clinical trial, the Philips IntelliSite Pathology Solution was the first WSI system to receive FDA approval. This first authorized de novo WSI predicate device has paved the way for other manufacturers who now only need 510(k) clearance instead of premarket approval. As a result, this has piqued the interest of many pathology laboratories that are now considering investing in WSI, and the digital pathology market has since witnessed a sizeable increase in pathology AI companies.
|Table 2: Regulatory milestones of whole-slide imaging in the United States|
Click here to view
| Vendor-Neutral Archives|| |
Despite its common use in conversations related to health and other images, the term vendor-neutral archive (VNA) has multiple definitions, none of which are widely accepted.,, As such, it is important that health-care systems use a greater degree of caution when purchasing a VNA to ensure that the purchased product has the desired functionality. While an ideal VNA in theory is one that provides a single software and hardware solution for all image viewing, storage, and retrieval, regardless of data type or source, the reality is that no VNA can currently work with all image types or acquisition sources. While in some circles a VNA is synonymous with a Picture Archiving and Communications System (PACS), VNAs are intended to be agnostic repositories of images and may not offer the worklist, annotation, or other functions often found with a PACS.
An enterprise VNA, composed of hardware and software, is used to accumulate images directly from various image acquisition sources including PACS and single-imaging devices. An enterprise VNA also has viewer capability for the collected images, and these viewers may be accessed via third-party end-user applications such as electronic health records (EHRs), laboratory information systems, and other health-related information systems and databases. Such viewing capability can also be extended to health information exchanges. [Figure 6] provides an example of an archetypal VNA configuration.
|Figure 6: An example of how a vendor-neutral archive might be integrated in a health system|
Click here to view
VNAs can be classified by how they are purchased, the functionalities of the viewer, and the type of image storage associated with them. A so-called true VNA is one that includes both storage and an integrated viewer. The viewer may come from the same or a different vendor, but this type of VNA is sold including both components, and the viewer is intended to be completely compatible with the associated image storage. A hybrid VNA is one in which the viewer and the image storage components must be purchased separately, in which the compatibility between the viewer and storage may have to be validated by the purchaser. Viewer functionalities are split into two general categories: those compatible with DICOM standards, also known as DICOM-compliant viewers, and viewers that are not DICOM compliant. DICOM-compliant viewers are also known as PACS-neutral archives and should be able to present images for viewing from any DICOM-compliant acquisition system. DICOM-compliant viewers may or may not be able to present non-DICOM images for viewing.
With regard to image storage, VNAs can have a central repository of images on hardware reserved for the VNA (central VNA), or the VNA may provide linking capability to images acquired and stored by third-party systems (federated VNA). Central VNAs facilitate processes for image backup, disaster recovery, and business continuity as well as interoperability with other systems such as outside organizations and health information exchanges. Central VNAs may additionally reduce the need for image data migration when new imaging systems, such as a radiology PACS, are implemented. While the cost of a central VNA can be quite steep, especially at the enterprise level, the cost of maintenance may be relatively low given its other advantages. While federated VNAs may be cheaper and faster to install because of their lower requirement for hardware infrastructure, establishing and maintaining image connectivity with a variety of image sources can be difficult, and many of the advantages of a central VNA will not be realized.
As VNAs have become more robust, some PACS have been deconstructed, meaning that some of the functions of the PACS have been offset to another application such as a VNA. Delegated functions can include image acquisition, workflow, interpretation, study management, and study access., One such function for which a VNA might be particularly well suited is image prefetch. Image prefetch is a workflow automation mechanism by which relevant prior studies of the patient are retrieved from image archives such as a VNA or PACS.
There are many potential advantages to VNAs., Centralizing all patient-related medical images into a single-viewing location creates enormous work efficiencies for end users. This not only creates increased provider satisfaction, but also has the potential to improve patient safety by making historical images easier to access and compare to current images. Access to historical images also reduces the risk of performing duplicate procedures or examinations, thereby reducing the potential for unnecessary radiation exposure, improving patient satisfaction, and decreasing overall health-care costs. The ability to view images across various departments would also improve patient care. Currently, many radiology and pathology departments have to go to extraordinary lengths to see corresponding images from the presurgical patient to the postsurgical specimen. Reducing the barriers to these images not only improves turnaround time for diagnoses where such comparisons are required for the diagnosis (e.g., orthopedic oncology), but it can also improve the chances of catching samples that have been mislabeled before they reach patients. Other patient care advantages have also been reported.,,
Creating a central repository of health images also presents new opportunities for patient engagement via patient portals, particularly for images that are not currently easily accessible in an EHR (e.g., pathology gross images). VNAs also provide new prospects for researchers as well as medical educators. Image acquisition in many areas of health care would also benefit from the improved security that some VNAs offer. This includes more robust authorization, authentication, and audit trailing of user activity. Better image security, backup, and redundancy procedures also prevent end users from moving or manipulating images such that other users are no longer able to access the original unmodified image.
However, like WSI applications, enterprise-level VNAs often do not get implemented because the hardware and software components are expensive, and its implementation can be difficult and/or time-consuming, especially if there are many systems required to integrate with the VNA. Similarly, if the imaging acquisition systems intended to integrate with the VNA lack adherence to standards such as DICOM or to image file format standards, many of which are de facto rather than governed by the International Standards Organization, then interoperability may be difficult or impossible. Lack of interoperability with the EHR was reported as a major hindrance to implementation in a recent survey of chief information officers., Workflow changes may also be large for some end users, and creating a sound business case may take a lot of time and resources. Data storage and memory usage are also areas of concern given the high rate of information acquisition.
Implementations of VNAs are expected to rise with enterprise-federated viewers. VNAs have been installed in about 40% of institutions surveyed., Independent of medical specialty, drivers of these installations have included, in decreasing order, improving care coordination (86%), reducing redundant testing (71%), and improving physician satisfaction (55%). In radiology centers, reducing radiation exposure for patients was also cited as beneficial (42%). The primary service lines connected to VNAs in this survey were radiology (91%) and cardiology (71%). Significantly lower connections were seen to service lines in dermatology (16%), gastroenterology (15%), and ophthalmology (10%). Pathology, despite its heavy use of both gross and photomicrographic images, was connected to VNAs in only 6% of institutions surveyed. Pathology departments utilize a wide range of image sources and types, and this is likely a contributing factor to pathology's low rate of connectivity to VNAs. While WSIs are becoming DICOM-compliant as newer equipment emerges, the vast majority of other types of images used in pathology are not DICOM compliant, thereby raising the bar for interoperability., Interoperability will require vendors to develop DICOM-compliant devices and software as well as capital expenditure on the part of laboratories to purchase interoperable components. Recent Connectathons conducted by the DICOM WG 26 held at Pathology Visions meetings and the European Digital Pathology Congress have shown that interoperability can be achieved, at least among the major vendors. Despite these demonstrations of interoperability, they have not yet translated into the pathology laboratory.
An enterprise approach to the selection and installation of a VNA is critical to its success. Most VNAs tend to get implemented when the radiology PACS is due to be replaced. Enterprise governance, strategic planning, stakeholder engagement, and paying attention to security issues (e.g., protected health information present in image metadata) are necessary components to successful implementation.,, A series of highly recommended articles about VNAs were produced by a collaborative effort between the Health Information Management and Systems Society and Society for Imaging Informatics in Medicine.,,,,,, Physician specialty groups who are less well known for imaging, including but not limited to pathologists, should not wait to approach their health-care systems about participating in enterprise VNA projects because, if properly implemented, these areas may greatly benefit from the use of a VNA. A sample list of information that institutions should consider gathering about each image category and device type acquired prior to selection of a VNA is shown in [Table 3].
|Table 3: Information to gather about each image category acquired or expected in the near future prior to engaging in a vendor-neutral archive implementation|
Click here to view
| Computational Pathology|| |
Researchers have worked in the area of pathology image analysis algorithm development for many years. Some algorithms focus on segmentation and/or classification of microanatomic objects such as cell nuclei, whereas others focus on generating automated or semi-automated image classifications. In many cases, analysis software produces collections of features corresponding to microanatomic objects (e.g., nuclei). Features describing each nucleus might consist of a polygon along with metrics quantifying size, shape, and texture. A detailed exposition of pathology image analysis algorithms is beyond the scope of this article, but excellent surveys of algorithmic work have been previously published.,,,,,,,,
The traditional approach to WSI characterization is to employ algorithms to extract imaging features. Features can be classified as being pixel-level, object-level, or semantic-level based. Pixel-level features are lowest in the information hierarchy, and examples of pixel-level features include mathematical characterizations of color, texture, and spatial patterns. Gray-level intensity profiles, Haralick Gray-level co-occurrence matrix features, wavelets, Gabor filter responses, and statistics and frequencies of color histograms are a small subset of pixel-level features. Object-level features are higher in the information hierarchy as they describe the characteristics of microanatomic objects such as nuclei, nucleoli, and mitoses, as well as more complex aggregates of multicellular microanatomic structures such as crypts, ducts, and blood vessels. Detailed characterization of object-level features typically requires segmenting the structure or structures in question. There is an extensive literature dedicated to documentation of methods employed to identify and segment nuclei along with substantial bodies of work dedicated to segmenting more complex aggregate structures. There is also extensive literature dedicated to object detection, where the location and relative position of structures are identified, but the detailed morphology of each object is not captured. Semantic-level features capture biological classification of microanatomic structures or regions. These features describe high-level concepts such as type of cell (e.g., epithelial or endothelial), presence or degree of dysplasia, presence of lymphocytes, and necrosis. Kothari et al. describe semantic-level features as being microanatomic objects with descriptive labels; one can also classify WSI regions in a semantic manner (e.g., a region might be classified as being a tumor region with lymphocyte infiltration).
Deep learning methods are rapidly making a major impact in digital pathology. These methods can be employed to identify and label objects, WSI regions, or to assign classifications to entire WSIs. Many researchers, including some of the authors of this article, have noted that deep learning methods tend to be more robust with less need to fine tune algorithm parameters and provide results that are relatively insensitive to variations in tissue staining and processing. However, systematic studies are needed to characterize the apparent robustness of many deep learning pathology methods. Feature extraction from WSIs often ensues in conjunction with clinical, molecular, and radiology data to predict cancer outcome and response to treatment. This work began in the early 2010s.,,, The Cancer Genome Atlas More Details provides an extraordinarily useful source of correlative data linking pathology WSIs with a rich set of molecular data, outcome and, in some cases, imaging data for varieties of cancer.
Computational pathology is increasingly seen as a key enabler for precision medicine. Major goals in the development of computational pathology-based biomarkers are to predict outcome and steer treatment. The notion is to integrate radiology and genomic biomarkers with computational pathology-based biomarkers in patient decision support. One example of the potential value of computational pathology lies in quantitative characterization of tumor-infiltrating lymphocytes (TILs). With the growth in cancer immunotherapy, quantitative and spatial characterization of the tumor microenvironment are likely to be of increasing clinical significance. While computational pathology is an extremely active research area, penetration of these methods into the clinical arena has been limited. The adoption of WSI for primary diagnosis will hopefully lead to subsequent clinical adoption of computational pathology methods over the coming decade.
| Large-Scale Data Management, Analysis, and Visualization|| |
Digital pathology has shown tremendous promise in helping advance our characterization of cancer, predict outcomes, and help steer treatment decisions. This has been aided by advances in image processing and machine learning coupled with an immense increase in available computational power. However, a major barrier to clinical adoption still involves technical challenges. These technical challenges can be broadly broken down into (1) data management, (2) systems and algorithms for analysis and visualization, and (3) visual analytics to aid in interpretation.
Data management of large WSI datasets is a key challenge. A single compressed WSI, scanned at ×40 magnification for a single focal plane, is typically 1–4 GB or even larger in size. Considering the volume of pathology slides produced at a typical institution, it is easy to see how the storage needs of digital pathology dwarf those of radiology. This requires novel storage systems that can rapidly access and retrieve large volumes of data. In addition to managing large volumes of images, another issue is the absence of a common image format. While a DICOM extension for digital pathology has been adopted in the DICOM specification, its adoption is only beginning to get traction. Currently, most WSI scanners are not DICOM compliant. The hope is that in the coming years, like in radiology, the community will converge around a common open image file format.
While image management is challenging, the management of imaging features is an equally daunting big data problem. Quantitatively extracted imaging features, better known as pathomic features, can range from a few dozen to up to half a million objects per image. Pathomic feature management requires supporting fast spatial queries that allow a user to pose a question like “extract nuclear features within areas of necrosis.” It is easy to see that feature management requires systems that can operate at large scales. Additionally, and more importantly, since computational pathology is such a rapidly evolving field, it requires a feature management system that is agile and can be easily adapted to the needs of the underlying biological problem. Thus, in place of standardized mechanisms for feature representation, a standardized application programming interface (API) is preferred. This approach has been adopted in the research genomics community – GA4GH – where the community collaborated and converged on a common API specification, and now competes on the underlying implementations. FeatureDB is an example of one such scalable and agile feature management system [Figure 7] that has been successfully used to manage features such as those that characterize nuclear material to region-based features such as TILs.
|Figure 7: An outline of FeatureDB for managing pathomic features and helping fuse clinical data and pathomic feature data for various analytical and visual data exploration and processing systems|
Click here to view
Image processing, machine learning, and, in recent years, deep learning algorithms have contributed immensely toward advancing the potential of digital pathology. These algorithms all rely on access to large volumes of high-quality labeled data. Given the large volume of data, it becomes cumbersome to move the data to algorithms for feature computation. Instead, systems must be designed to enable the movement of algorithms to the data and where possible collocate the algorithms and data. Containerization and the increasing popularity of systems like Docker have made it possible to encapsulate algorithms, deploy them at scale on a public/private cloud, and compute with minimal image movement. This is an area of active research, where the imaging community can learn and adopt many tools from the genomics community.
Visualization and visual analytics are a key component of digital pathology. Image visualization, or the ability to rapidly view and explore a digitized WSI via pan-and-zoom tool, is essential. One requirement is the ability to overlay images with features to aid in the interpretability of pathomic feature sets. In addition to overlaying features on images, users need visual analytic tools for population-wide exploration of the feature space. This would allow, for example, examining the correlations among the imaging features for a particular cohort or one could employ such tools to examine the interplay between imaging features and corresponding mutation information and/or survival outcomes [Figure 8]. Quantitative imaging informatics for pathology is a comprehensive technological stack that supports such rich visualization and visual analytic tools.
|Figure 8: Scientific mashups and visual analytics for interactive exploration of fused rad/path/clinical data. Such visual explorations help researchers examine the scope and extent of data, create specific cohorts, and formulate hypotheses|
Click here to view
| Conclusion|| |
The last 20 years have had significant triumphs related to WSI. Users today have a plethora of commercial WSI scanners to select from both clinical and nonclinical uses. Major advances in technology have been witnessed related to hardware (e.g., z-scanning ability, hybrid WSI/robotic instruments) and software tools (e.g., image analysis). While FDA approval of the WSI system for primary diagnosis in surgical pathology is anticipated to promote greater adoption of WSI for clinical use in the United States, there are still a number of questions that need to be addressed.
Pathology, like many other medical specialties, stands to benefit from a successfully implemented VNA. Although an enterprise VNA will benefit from the inclusion of pathology images, the lack of standardization and universal DICOM compliance of WSIs is expected to delay the integration of these images into these emerging applications. Such integration will be made easier with appropriate planning, thorough inventory assessment, and appropriate selection of VNA and image acquisition products.
The next 20 years promises to see much greater use of digital pathology, thereby enhancing the pathologists' ability to deliver patient care. The implementation of meaningful ways to validate, classify, and explore pathology imaging biomarkers integrated into the clinical decision-making process will be a key pathology informatics contribution to precision medicine. Understanding the interplay between morphology and molecular mechanisms is central to the success of research targeting practically every major disease. Digital pathology enables many of these studies. Indeed, a number of research groups have developed and demonstrated a rich set of methods including deep learning for carrying out quantitative microscopy analyses. However, to bring this vision to reality, it will be necessary to develop and deploy infrastructure to scan, catalog, and store extremely large collections of WSI, both for clinical work and research. In conclusion, although the concept of the virtual microscope is two decades old, the adoption of digital pathology informatics tools in clinical practice is clearly still a work in progress.
U24CA180924, U24CA215109. The authors would like to thank Andrew Lesniak for his help regarding the commercial history of digital pathology.
Financial support and sponsorship
Conflicts of interest
Liron Pantanowitz is on the medical advisory board for Leica and Ibex and serves as a consultant for Hamamatsu.
| References|| |
Pantanowitz L. Digital images and the future of digital pathology. J Pathol Inform 2010;1. pii: 15.
Pantanowitz L. Whole slide imaging. In: Pantanowitz L, Parwani AV, editors. Digital Pathology. Chicago, USA: ASCP Press; 2017. p. 59-76.
Chang C, Moon B, Acharya A, Shock C, Sussman A, Saltz J. Titan: A high-performance remote-sensing database. In: Data Engineering. Proceedings. IEEE 13th
International Conference; 1997. p. 375-84.
Ferreira R, Moon B, Humphries J, Sussman A, Saltz J, Miller R, et al.
The virtual microscope. Proc AMIA Annu Fall Symp 1997:449-53.
Afework A, Beynon MD, Bustamante F, Cho S, Demarzo A, Ferreira R, et al.
Digital dynamic telepathology – The virtual microscope. Proc AMIA Symp 1998:912-6.
Catalyürek U, Beynon MD, Chang C, Kurc T, Sussman A, Saltz J, et al.
The virtual microscope. IEEE Trans Inf Technol Biomed 2003;7:230-48.
Kurc T, Catalyurek U, Chang C, Sussman A, Saltz J. Visualization of large data sets with the active data repository. IEEE Comput Graph Appl 2001;21:24-33.
Beynon M, Ferreira R, Kurc T, Sussman A, Saltz J. DataCutter: Middleware for filtering very large scientific datasets on archival storage systems. IEEE Symposium on Mass Storage Systems; 2000. p. 119-34.
Aji A, Wang F, Vo H, Lee R, Liu Q, Zhang X, et al.
Hadoop-GIS: A High performance spatial data warehousing system over MapReduce. Proceedings VLDB Endowment 2013;6. pii: p1009.
Baig F, Mehrotra M, Vo H, Wang F, Saltz J, Kurc T. SparkGIS: Efficient comparison and evaluation of algorithm results in tissue image analysis studies. In: Wang F, Luo G, Weng C, Khan A, Mitra P, Yu C, editors. Biomedical Data Management and Graph Online Querying. Big-O(Q) 2015, DMAH 2015. Lecture Notes in Computer Science. Vol. 9579. Cham: Springer; 2016.
Goode A, Gilbert B, Harkes J, Jukic D, Satyanarayanan M. OpenSlide: A vendor-neutral software foundation for digital pathology. J Pathol Inform 2013;4:27.
] [Full text]
Singh R, Chubb L, Pantanowitz L, Parwani A. Standardization in digital pathology: Supplement 145 of the DICOM standards. J Pathol Inform 2011;2:23.
] [Full text]
Gutman DA, Khalilia M, Lee S, Nalisnik M, Mullen Z, Beezley J, et al.
The digital slide archive: A software platform for management, integration, and analysis of histology for cancer research. Cancer Res 2017;77:e75-8.
Martel AL, Hosseinzadeh D, Senaras C, Zhou Y, Yazdanpanah A, Shojaii R, et al.
An image analysis resource for cancer research: PIIP-pathology image informatics platform for visualization, analysis, and management. Cancer Res 2017;77:e83-6.
Saltz J, Sharma A, Iyer G, Bremer E, Wang F, Jasniewski A, et al.
Acontainerized software system for generation, management, and exploration of features from whole slide tissue images. Cancer Res 2017;77:e79-82.
Bankhead P, Loughrey MB, Fernández JA, Dombrowski Y, McArt DG, Dunne PD, et al.
QuPath: Open source software for digital pathology image analysis. Sci Rep 2017;7:16878.
Bacus JV, Bacus JW. Method and Apparatus for Acquiring and Reconstructing Magnified Specimen Images from a Computer-Controlled Microscope. US Patent 6,101,265; 2000.
Pantanowitz L, Sinard JH, Henricks WH, Fatheree LA, Carter AB, Contis L, et al.
Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch Pathol Lab Med 2013;137:1710-22.
Elgarby E, Parwani AV, Pantanowitz L. Whole slide imaging: Widening the scope of cytopathology. Diagn Histopathol 2014;20:456-61.
Nielsen PS, Lindebjerg J, Rasmussen J, Starklint H, Waldstrøm M, Nielsen B, et al.
Virtual microscopy: An evaluation of its validity and diagnostic performance in routine histologic diagnosis of skin tumors. Hum Pathol 2010;41:1770-6.
Al-Janabi S, Huisman A, Vink A, Leguit RJ, Offerhaus GJ, ten Kate FJ, et al.
Whole slide images for primary diagnostics of gastrointestinal tract pathology: A feasibility study. Hum Pathol 2012;43:702-7.
Al-Janabi S, Huisman A, Vink A, Leguit RJ, Offerhaus GJ, Ten Kate FJ, et al.
Whole slide images for primary diagnostics in dermatopathology: A feasibility study. J Clin Pathol 2012;65:152-8.
Snead DR, Tsang YW, Meskiri A, Kimani PK, Crossman R, Rajpoot NM, et al.
Validation of digital pathology imaging for primary histopathological diagnosis. Histopathology 2016;68:1063-72.
Saco A, Ramírez J, Rakislova N, Mira A, Ordi J. Validation of whole-slide imaging for histolopathogical diagnosis: Current state. Pathobiology 2016;83:89-98.
Williams BJ, DaCosta P, Goacher E, Treanor D. A systematic analysis of discordant diagnoses in digital pathology compared with light microscopy. Arch Pathol Lab Med 2017;141:1712-8.
Bongaerts O, Clevers C, Debets M, Paffen D, Senden L, Rijks K, et al.
Conventional microscopical versus digital whole-slide imaging-based diagnosis of thin-layer cervical specimens: A validation study. J Pathol Inform 2018;9:29.
] [Full text]
Stathonikos N, Veta M, Huisman A, van Diest PJ. Going fully digital: Perspective of a Dutch academic pathology lab. J Pathol Inform 2013;4:15.
] [Full text]
Cheng CL, Azhar R, Sng SH, Chua YQ, Hwang JS, Chin JP, et al.
Enabling digital pathology in the diagnostic setting: Navigating through the implementation journey in an academic medical centre. J Clin Pathol 2016;69:784-92.
Fraggetta F, Garozzo S, Zannoni GF, Pantanowitz L, Rossi ED. Routine digital pathology workflow: The Catania experience. J Pathol Inform 2017;8:51.
] [Full text]
Williams BJ, Bottoms D, Treanor D. Future-proofing pathology: The case for clinical adoption of digital pathology. J Clin Pathol 2017;70:1010-8.
Abels E, Pantanowitz L. Current state of the regulatory trajectory for whole slide imaging devices in the USA. J Pathol Inform 2017;8:23.
] [Full text]
Mukhopadhyay S, Feldman MD, Abels E, Ashfaq R, Beltaifa S, Cacciabeve NG, et al.
Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: A Multicenter blinded randomized noninferiority study of 1992 cases (Pivotal study). Am J Surg Pathol 2018;42:39-52.
Agarwal TK, Sanjeev. Vendor neutral archive in PACS. Indian J Radiol Imaging 2012;22:242-5.
] [Full text]
Medical Imaging & Technology Alliance (A Division of the National Electrical Manufacturers Association). The DICOM Standard. Rosslyn, VA: NEMA; 2016.
Forsberg D, Rosipko B, Sunshine JL, Ros PR. State of integration between PACS and other IT systems: A National survey of academic radiology departments. J Am Coll Radiol 2016;13:812-8.e2.
Harwalkar K. DICOM Prefetch: Quick access to priors. The DICOM 2013 International Conference & Seminar. Bangalore, India; 15 March, 2013.
Strowig G. Making the switch to vendor-neutral archiving. How to optimize comprehensive data storage for healthcare's new age. Health Manag Technol 2013;34:10, 12-3.
Reeves K. Case study: Collaborative care. In: Imaging 3.0. Reston, VA: American College of Radiology; 2016. p. 1-3.
LifeIMAGE. CIO Perspectives on Enterprise Imaging: A Survey of CHIME Members Explores the Drivers, Challenges and Benefits of Image Data Interoperability; 2016. Available from: http://info.lifeimage.com/chime-ebook
. [Last accessed on 2018 Nov 12].
Murphy K. Lack of EHR Interoperability Holding Back Enterprise Imaging. In. Health IT Interoperability; 2016.
Miliard M. Storage Wars: As Data has Proliferated These Past 10 Years, Providers have had to Figure out Where to Put it. In. Healthcare IT News; 2013.
Rasband M, Chistensen J, Paxman E, Smith B, Rasmussen A, Zidel J. Enterprise Imaging 2015: Strategies, Options, and Vendor Performance; December, 2015.
National Electrical Manufacturers Association. Digital Imaging and Communications in Medicine (DICOM): Supplement 122: Specimen Module and Revised Pathology SOP Classes. Rosslyn, VA: National Electrical Manufacturers Association; 2008. p. 1-78.
National Electrical Manufacturers Association. Digital Imaging and Communications in Medicine (DICOM): Supplement 145: Whole Slide Microscopic Image IOD and SOP Classes. Rosslyn, VA: National Electrical Manufacturers Association; 2010. p. 1-59.
Branz K. Transitioning to a vendor-neutral image archive. Health Manag Technol 2013;34:16-7.
Roth CJ, Lannum LM, Joseph CL. Enterprise imaging governance: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:539-46.
Clunie DA, Dennison DK, Cram D, Persons KR, Bronkalla MD, Primo HR, et al.
Technical challenges of enterprise imaging: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:583-614.
Cram D, Roth CJ, Towbin AJ. Orders- versus encounters-based image capture: Implications pre- and post-procedure workflow, technical and build capabilities, resulting, analytics and revenue capture: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:559-66.
Roth CJ, Lannum LM, Dennison DK, Towbin AJ. The current state and path forward for enterprise image viewing: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:567-73.
Roth CJ, Lannum LM, Persons KR. A foundation for enterprise imaging: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:530-8.
Towbin AJ, Roth CJ, Bronkalla M, Cram D. Workflow challenges of enterprise imaging: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:574-82.
Vreeland A, Persons KR, Primo HR, Bishop M, Garriott KM, Doyle MK, et al.
Considerations for exchanging and sharing medical images for improved collaboration and patient care: HIMSS-SIIM collaborative white paper. J Digit Imaging 2016;29:547-58.
Kothari S, Phan JH, Stokes TH, Wang MD. Pathology imaging informatics for quantitative analysis of whole-slide images. J Am Med Inform Assoc 2013;20:1099-108.
Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B, et al.
Histopathological image analysis: A review. IEEE Rev Biomed Eng 2009;2:147-71.
Ghaznavi F, Evans A, Madabhushi A, Feldman M. Digital imaging in pathology: Whole-slide imaging and beyond. Annu Rev Pathol 2013;8:331-59.
Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: Challenges and opportunities. Med Image Anal 2016;33:170-5.
Janowczyk A, Madabhushi A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J Pathol Inform 2016;7:29.
] [Full text]
Irshad H, Veillard A, Roux L, Racoceanu D. Methods for nuclei detection, segmentation, and classification in digital histopathology: A review-current status and future potential. IEEE Rev Biomed Eng 2014;7:97-114.
Xing F, Yang L. Robust nucleus/Cell detection and segmentation in digital pathology and microscopy images: A comprehensive review. IEEE Rev Biomed Eng 2016;9:234-63.
Hamilton PW, Bankhead P, Wang Y, Hutchinson R, Kieran D, McArt DG, et al.
Digital pathology and image analysis in tissue biomarker research. Methods 2014;70:59-73.
Al-Janabi S, Huisman A, Van Diest PJ. Digital pathology: Current status and future perspectives. Histopathology 2012;61:1-9.
Murthy V, Hou L, Samaras D, Kurc TM, Saltz JH. Center-focusing multi-task CNN with injected features for classification of glioma nuclear images. IEEE Winter Conf Appl Comput Vis 2017;2017:834-41.
Al-Milaji Z, Ersoy I, Hafiane A, Palaniappan K, Bunyak F. Integrating segmentation with deep learning for enhanced classification of epithelial and stromal tissues in H&E images. Pattern Recognit Lett 2017:1-8. [doi.org/10.1016/j.patrec. 2017.09.015].
Hou L, Samaras D, Kurc TM, Gao Y, Davis JE, Saltz JH, et al.
Patch-based convolutional neural network for whole slide tissue image classification. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2016;2016:2424-33.
Cooper LA, Kong J, Wang F, Kurc T, Moreno CS, Brat DJ, et al.
Morphological signatures and genomic correlates in glioblastoma. Proc IEEE Int Symp Biomed Imaging 2011:1624-7.
Beck AH, Sangoi AR, Leung S, Marinelli RJ, Nielsen TO, van de Vijver MJ, et al.
Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med 2011;3:108ra113.
Yu KH, Zhang C, Berry GJ, Altman RB, Ré C, Rubin DL, et al.
Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat Commun 2016;7:12474.
Luo X, Zang X, Yang L, Huang J, Liang F, Rodriguez-Canales J, et al.
Comprehensive computational pathological image analysis predicts lung cancer prognosis. J Thorac Oncol 2017;12:501-9.
Cooper LA, Demicco EG, Saltz JH, Powell RT, Rao A, Lazar AJ, et al.
PanCancer insights from the cancer genome atlas: The pathologist's perspective. J Pathol 2018;244:512-24.
Evans AJ, Bauer TW, Bui MM, Cornish TC, Duncan H, Glassy EF, et al.
US food and drug administration approval of whole slide imaging for primary diagnosis: A Key milestone is reached and new questions are raised. Arch Pathol Lab Med 2018;142:1383-7.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7], [Figure 8]
[Table 1], [Table 2], [Table 3]