Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 598  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 

Table of Contents    
J Pathol Inform 2020,  11:22

A regulatory science initiative to harmonize and standardize digital pathology and machine learning processes to speed up clinical innovation to patients

1 Department of Pathology, Center for Integrated Diagnostics, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA
2 Division of Imaging, Diagnostics, and Software Reliability, Center for Devices and Radiological Health, Food and Drug Administration, Office of Science and Engineering Laboratories, Silver Spring, MD, USA
3 Visiopharm Americas, Westminster, CO, USA
4 Department of Pathology, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA
5 Hamamatsu Corporation, Pittsburgh, PA, USA
6 LUMEA, Salt Lake City, UT, USA
7 Department of Pathology and Immunology, Washington University School of Medicine, St. Louis, MO, USA
8 Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
9 Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA
10 Medical Device Innovation Consortium, Arlington, VA, USA
11 Roche Tissue Diagnostics, Santa Clara, USA
12 Friends of Cancer Research, Washington, DC, USA
13 PathAI, Boston, MA, USA

Date of Submission01-Apr-2020
Date of Decision20-Apr-2020
Date of Acceptance16-Jun-2020
Date of Web Publication06-Aug-2020

Correspondence Address:
Dr. Jochen K Lennerz
Department of Pathology, Center for Integrated Diagnostics, Harvard Medical School, Massachusetts General Hospital, 55 Fruit Street, GRJ1015 Boston, MA 02114
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/jpi.jpi_27_20

Rights and Permissions

Unlocking the full potential of pathology data by gaining computational access to histological pixel data and metadata (digital pathology) is one of the key promises of computational pathology. Despite scientific progress and several regulatory approvals for primary diagnosis using whole-slide imaging, true clinical adoption at scale is slower than anticipated. In the U.S., advances in digital pathology are often siloed pursuits by individual stakeholders, and to our knowledge, there has not been a systematic approach to advance the field through a regulatory science initiative. The Alliance for Digital Pathology ( the Alliance) is a recently established, volunteer, collaborative, regulatory science initiative to standardize digital pathology processes to speed up innovation to patients. The purpose is: (1) to account for the patient perspective by including patient advocacy; (2) to investigate and develop methods and tools for the evaluation of effectiveness, safety, and quality to specify risks and benefits in the precompetitive phase; (3) to help strategize the sequence of clinically meaningful deliverables; (4) to encourage and streamline the development of ground-truth data sets for machine learning model development and validation; and (5) to clarify regulatory pathways by investigating relevant regulatory science questions. The Alliance accepts participation from all stakeholders, and we solicit clinically relevant proposals that will benefit the field at large. The initiative will dissolve once a clinical, interoperable, modularized, integrated solution (from tissue acquisition to diagnostic algorithm) has been implemented. In times of rapidly evolving discoveries, scientific input from subject-matter experts is one essential element to inform regulatory guidance and decision-making. The Alliance aims to establish and promote synergistic regulatory science efforts that will leverage diverse inputs to move digital pathology forward and ultimately improve patient care.

Keywords: Artificial intelligence, digital pathology, machine learning, regulatory science, slide scanning

How to cite this article:
Marble HD, Huang R, Dudgeon SN, Lowe A, Herrmann MD, Blakely S, Leavitt MO, Isaacs M, Hanna MG, Sharma A, Veetil J, Goldberg P, Schmid JH, Lasiter L, Gallas BD, Abels E, Lennerz JK. A regulatory science initiative to harmonize and standardize digital pathology and machine learning processes to speed up clinical innovation to patients. J Pathol Inform 2020;11:22

How to cite this URL:
Marble HD, Huang R, Dudgeon SN, Lowe A, Herrmann MD, Blakely S, Leavitt MO, Isaacs M, Hanna MG, Sharma A, Veetil J, Goldberg P, Schmid JH, Lasiter L, Gallas BD, Abels E, Lennerz JK. A regulatory science initiative to harmonize and standardize digital pathology and machine learning processes to speed up clinical innovation to patients. J Pathol Inform [serial online] 2020 [cited 2021 Nov 27];11:22. Available from:

   Introduction Top

“The scientist and science provide the means, the politician and politics decide the ends.”

-Alvin M. Weinberg[1]

Regulatory science is an established discipline that entails the application of the scientific method to support regulatory and other policy objectives.[2] Simply put, when medical research provides a novel solution to a health need, regulatory science applies the scientific method to assess benefits and risks before marketing for clinical use. To assess benefits and risks, regulatory scientists develop new tools, standards, and approaches to evaluate the effectiveness, safety, and quality of medical products. A primary challenge in the field of digital pathology is the lack of understanding that strong relationships between regulatory, basic, and translational scientists can substantially improve clinical innovation.[3],[4],[5],[6] For example, regulatory science is not restricted to regulatory agencies.[2],[4],[5],[6] As a scientific discipline, regulatory science challenges current concepts of benefit and risk assessments, submission and approval strategies, patient involvement, and various ethical aspects. Regulatory science includes the creation of a scientific dialog for launching new ideas – not only derived from industry and regulatory authorities but also by, for example, academics, clinicians, and patients.[7] It has been recognized that regulatory science can have a significant impact in bringing new devices to patients in need.[7]

Here, we outline a recently established, volunteer, collaborative regulatory science initiative termed the Alliance for Digital Pathology (the Alliance). To prevent confusion, our intent is to familiarize the community with the aims, scope, and rationale of the Alliance. The Alliance aims to move the field of digital pathology forward by systematically assessing relevant aspects and providing publicly available resources (e.g., data, tools, and methods) to inform and improve the relevant regulatory guidance landscape.[8] Our premise (thesis) is that the Alliance promotes regulatory science as a bridge between digital pathology (the means) and moving the field of diagnostic pathology forward (the ends). By promoting regulatory science, the Alliance helps to unlock the potential of new technologies and thereby overcomes the dichotomy illustrated in the epigraph by Dr. Weinberg.[1]

   Toward an Operational Definition of a Clinical, Interoperable, Integrated Solution for Digital Pathology Top

The key aim of the Alliance is to help convert the existing (traditional) pathology technologies and workflows into interoperable, digitally enhanced solutions by contributing regulatory science deliverables that can be used to inform and improve the applicable regulatory guidance landscape. Numerous groups have attempted to specify the relevant components of digital pathology solutions;[9],[10],[11],[12],[13],[14],[15],[16],[17],[18] however, given the modularized nature of diagnostic pathology, defining the specific scope of a digital pathology solution is highly context dependent. For example, the variability of a stain (e.g., hematoxylin and eosin across or within laboratories) may influence the performance of a downstream mutation prediction algorithm.[19],[20],[21] In this example, one may consider drawing an arbitrary boundary before the staining step; however, the fixation and processing method (e.g., formalin fixed, paraffin embedded) or even the tissue acquisition, handling, or image acquisition[22] may influence the performance of the predictor as well. Thus, for the purpose of the Alliance, we considered three descriptors for the solution. First, we aim toward a clinical (as opposed to a research-based) solution. Second, due to the modularized nature of the various subprocesses within the main workflows in pathology, we aim for interoperability of systems. Third, to account for the various and arbitrary boundaries of workflow steps (modules) and technologies relevant for a given task (intended use), we consider every step, from the medical procedure acquiring the cell or tissue sample all the way to the fully integrated diagnostic output (e.g., report or model output), as relevant. As opposed to an end-to-end solution, where the supplier of an application or system will provide all the hardware and/or software to meet specific requirements, we are aiming for modularized solutions within the main workflow. We refer to these three solution descriptors (clinical, interoperable, and modularized) as an “integrated solution for digital pathology. We acknowledge that this definition is operational and arguably incomplete yet represents a technique that enables flexible modeling to solve challenging problems.[23],[24],[25],[26]

   The Multifaceted Nature of Digital Pathology Needs Increased Regulatory Clarity Top

Digital pathology has grown into a multimillion-dollar vendor landscape,[27] and the application of machine learning algorithms holds big promise for improving diagnostics in numerous ways.[28],[29],[30] Despite this active and promising research, the Food and Drug Administration (FDA) has only recently authorized two digital pathology whole-slide imaging (WSI) systems for primary diagnosis.[3],[9],[11],[31],[32] Even with the authorization of two WSI systems and numerous use cases,[12],[13],[14],[18],[33],[34],[35],[36],[37],[38] in the U.S., we see few hospitals changing their daily clinical operations to integrate WSI for primary diagnosis.[39],[40],[41],[42],[43] Clinical laboratories face additional challenges when implementing high complexity and/or high-risk medical devices coupled with software solutions as laboratory-developed tests (LDTs).[44],[45],[46] For example, even when using an FDA-authorized whole-slide imaging device, the approval or clearance does not eliminate the need for an individual laboratory to verify the performance of these systems for the specific intended diagnostic purpose. Specifically, Clinical Laboratory Improvement Amendments of 1988 or CLIA '88 in the US requires at least verification[47] and substantial adaptation to implement.[48],[49],[50],[51],[52]

One value proposition for digital pathology is to take advantage of the digital nature of WSI and use artificial intelligence/machine learning (AI/ML) algorithms to support clinical decisions.[11],[53] In fact, several groups have proposed that AI/ML will unlock the full potential of digital pathology.[53],[54]

To examine the current regulatory guidance landscape related to digital pathology and AI, four authors (HDM, RH, EA, and JKL) performed a review of pertinent documents from the FDA. We noted the official release dates and assigned each document to one of five dimensions [Figure 1] and [Supplemental Table 1 [Additional file 1]]. By plotting these documents and dimensions over time, we show how the regulatory guidance landscape evolves. A novice in the field may look for one comprehensive guidance document for digital pathology and may be discouraged by the initial complexity; however, we hope that [Figure 1] provides a reasonable starting point for learning the current regulatory guidance landscape. As we show [Figure 1], arrows], the regulatory guidance landscape adapts over time as technologies and the associated regulatory science matures. One key element in the multistep process to improve the regulatory guidance landscape is critical scientific input from subject-matter experts.[3],[4],[5],[10],[11],[15],[53] We strongly believe that “watching and waiting” will not help the case of digital pathology. Similarly, workarounds[84],[85],[86],[87],[88],[89] turn into long and winding roads that ultimately end at the FDA and within the FDA's regulatory framework.[83] The Alliance intends to organize subject-matter experts and provide scientific input.
Figure 1: Overview of selected FDA guidance documents. Four of the authors (HM, RH, EA, and JKL) performed a meta-review of selected FDA guidance documents relevant to the scope and aims of the Alliance. The figure shows grouping of these guidance documents across five dimensions over time. Please note: the numbers refer to the order of review during the meta-review process; Supplemental Table 1 provides the original release dates, the official FDA guidance title, and the issuer. AI/ML: Artificial intelligence/machine learning; CMS: Centers for Medicare and Medicaid Services; FDA: Food and Drug Administration; IMDRF: International Medical Device Regulators Forum; MDDT: Medical Device Development Tools; SaMD: Software as a Medical Device; QMS: Quality management system; WSI: Whole-slide imaging

Click here to view

Simply put, the practical dilemma in digital pathology is that developers are challenged to create an FDA submission following the evolving and complex regulatory guidance landscape, and the adoption of WSI by pathologists is slowed because they cannot realize the full potential and utility of digital pathology and AI/ML without full clinical integration. The field of digital pathology is looking for broader guidance, practical advice, and streamlined regulatory pathways to help navigate this uncharted and exciting territory.

   Regulatory Science, the Precompetitive Space, and Real-World Evidence Top

FDA clearance of a medical device offers a vendor market access. Once introduced, market forces tend not to encourage the vendor to make the device or its subsystems interoperable.[55],[56],[57],[58],[59],[60],[61] We like to emphasize that routine diagnostic pathology is highly modularized and the practice does not lend itself easily to nonmodular, locked down solutions.[3],[9],[10],[11],[27],[50],[51],[54],[62] The Alliance believes that it can promote interoperability and innovation by launching initiatives and creating deliverables (data, standards, tools, and methods) in the precompetitive space. Organizing industry to work collaboratively in the precompetitive space will eliminate unnecessary or duplicative (proprietary) efforts and thereby save all parties' time, money, and resources when pursuing device authorizations.[63] The Alliance initiatives and deliverables will speed clinical integration and carry mutual benefit to all stakeholders, including regulators, clinicians, manufacturers, and most importantly, patients.

Real-world evidence (RWE) comes from the competitive, postmarket space. RWE can identify trends in adverse events, summarize where resources are being spent, and track the impact of a new diagnostic device or therapy in terms of patient outcomes. RWE can support clinical practice guidelines and decisions about reimbursement and policy. Furthermore, RWE can inform regulatory decision making, as effectively demonstrated by the Medical Device Innovation Consortium,[64],[65] the National Evaluation System for health Technology Coordinating Center,[66] the Patient-Centered Outcomes Research Institute,[67],[68] Friends of Cancer Research,[69],[70] and others.[3],[5],[6],[9],[71],[72],[73],[74]

   From Key Mission Elements to a Delivery Process Top

Accomplishing mutual benefit to multiple stakeholders is a daunting value proposition that requires a unique regulatory science approach and stakeholder involvement for selection and prioritization of deliverables. The approach of the Alliance [Figure 2]a is to deliver tools by harnessing existing, precompetitive FDA programs and use the gained experience to inform effective regulation. The approach thereby aims to streamline precompetitive and eventually competitive submissions that enable faster time to market to improve patient care. Regulatory science deliverables, including tools and the experience from precompetitive submissions, will be shared, and when one integrated solution has been enabled, the Alliance can dissolve [Figure 2]a. The key mission elements of the Alliance are summarized in [Table 1].[75]
Figure 2: Concept, process, role, and proposed benefits of the Alliance. (a) The approach of the Alliance is to deliver tools via precompetitive FDA programs and use the gained experience to support effective FDA review. The concept also includes a predetermined exit strategy (i.e., one fully integrated solution for digital pathology). (b) The process of moving Alliance projects forward is essentially a two-step, multidisciplinary peer review by subject-matter experts. First, projects are reviewed, and after a multidisciplinary selection process that emphasizes the patient perspective and relevance for patient care, the steering committee (jointly with relevant partners) attempts to allocate resources. (c) Role and proposed benefits of the Alliance exemplified using the high-throughput truthing project for tumor-infiltrating lymphocytes as a biomarker in breast cancer. AMCs: Academic medical centers; MDDT: Medical Device Development Tools (precompetitive FDA submission program); Mock: mock submission program (precompetitive FDA submission program); OIR: Office of In vitro Diagnostics and Radiological Health; OPEQ: Office of Product Evaluation and Quality; OSEL: Office of Science and Engineering Laboratories; FDA: Food and Drug Administration

Click here to view
Table 1: Key mission elements of the Alliance

Click here to view

To align stakeholder interests, initiatives and deliverables need to be prioritized and prioritization requires a process. We conceptualized an approach that is composed of synergistic review, project components, and resource allocation [Figure 2]b. The process starts with synergizing various stakeholder interests into concise individual projects. An Alliance project may consist of a clinically relevant intended use case, a data set (e.g., pixel and metadata), and an applicable regulatory science pathway [e.g., [Figure 2]b, triangle]. The Alliance membership, composed of subject-matter experts from various domains, will have the opportunity to review, contribute, and potentially modify these projects through free and voluntary feedback to the project owner. Over time, individual effort and maturation of ideas will result in optimized projects (“big ideas”). To help realize the proposed deliverables and/or allocate additional resources, we established the Alliance Steering Committee, a flexible organizational structure, and a code of conduct [Supplemental Table 2 [Additional file 2]]

An example project is illustrated in [Figure 2]c. A subset of members in the Alliance are studying the relevance of tumor-infiltrating lymphocytes (TILs) as a prognostic and predictive biomarker.[76],[77] The interest in this clinical use case led to a collaborative project that includes members from the FDA, academic medical centers (AMCs), and industry. The project, referred to as the high-throughput truthing (HTT) project, aims to demonstrate the collection and use of pathologist annotations for the purpose of evaluating AI/ML algorithms and other digital pathology initiatives. The project also aims to qualify the glass slides, whole-slide images, and pathologist annotations for evaluating AI/ML algorithms through the precompetitive FDA's Medical Device Development Tools (MDDT) program.[78] If qualified, the “ground-truth” materials can serve as a publicly available, standardized evaluation “tool” for algorithm evaluation that can be used in submissions to the FDA.

In relation to the Alliance, the HTT project was submitted to the Alliance and discussed in November 2019. The Alliance can contribute in multiple ways to accelerate the realization of this and similar projects. First, the Alliance confirmed that the aims of the project could benefit many stakeholders.

The discussions provided useful feedback from subject-matter experts regarding the clinical use case, sourcing slides from multiple sites, agreements for sharing materials within the project, and issues related to sharing materials publicly. The discussions also identified future work that could build on the lessons, methods, infrastructure, and relationships created while pursuing the current aims. Important future work identified in the discussions included scaling the effort to address generalizability across sites and generalizability across use cases.

The Alliance has since provided help with the project [Figure 2]b, triangle 01, relevant intended use case; [Figure 2]c, 01] by disseminating the project needs. This networking through the Alliance has yielded volunteers for sourcing and scanning slides, pathologists to annotate slides and images, and opportunities to collect data. Connections have been created that are expected to help in the development of the statistical analyses and the future hosting of slides, images, and annotations. Currently, the project is developing the strategy and materials for the FDA's MDDT program [Figure 2]b, triangle, MDDT; [Figure 2]c, 03]. The development is a learning experience for all involved, with contributions from project and Alliance subject-matter and regulatory affairs experts. The learning experience is expected to continue through official interactions with the FDA related to the MDDT submission. Thus, aside from helping to create the ground-truth data set, the Alliance aims to understand regulatory issues and processes for future streamlining of other projects and submissions. As demonstrated here, a qualified data set may result in time-savings when preparing submissions, generating additional tools, and streamlining regulatory review, resulting in faster time to market and improved patient care.

   Who is the Alliance ? Top

The Alliance is composed of a diverse and interdisciplinary group of stakeholders who contribute to various aspects of diagnostic pathology, from tissue acquisition to reporting and data analytics. When deconstructing the clinical digital pathology and AI/ML pipeline into its component parts, numerous workflow steps have to function in unison [Figure 3]a. Aside from the modular nature and operational complexity, these components emphasize the importance of involving various stakeholders with each module. Given the novelty of pursuing a collaborative regulatory science effort to solve the challenge of clinical adoption of digital pathology, we noted a lack of coRegulatory Submissions

ncrete data on interested stakeholders and their priorities. In September 2019, we conducted an internal survey [n = 42; [Supplemental Table 3 [Additional file 3]]. At that time, the survey respondents stated that the top 3 deliverables/workflow steps to focus on should be the DICOM standard, AI/ML test validation, and pixel and metadata capture [Figure 3]b. By self-reported primary affiliation, the Alliance encompasses representation from academia (32%), industry (50%), government regulators and nongovernment organizations (12%), and patient advocacy groups (6%) [Figure 3]c.
Figure 3: Workflow steps and Alliance survey results. (a) Digital pathology workflows include preanalytical, retrieval, scan (image acquisition), clinical data, metadata, machine learning algorithm development, clinical integration, clinical utility, and financial sustainability considerations; all dependent on the specific use case/application. These workflow steps correspond to the axis labels in Figure 3b. (b) The Alliance conducted a survey among the members in September 2019. Bar graphs show the workflow steps that survey respondents felt the Alliance should focus on. These steps are reflected in a workflow diagram in Figure 3a. (c) Survey results from September 2019. DICOM: Digital Imaging and Communications in Medicine (here referring to an interoperable file format for digital pathology); EHR: Electronic health record; H&E: Hematoxylin and eosin stain; IHC: Immunohistochemistry; LIMS: Laboratory information management system; MDIC: Medical Device Innovation Consortium

Click here to view

   Meetings, Growth, and Working Groups Top

Since its inception in May 2019, the Alliance hosted numerous teleconferences, web meetings, and three, in-person, national meetings [Figure 4]a. Over this period (May 2019–January 2020), the Alliance membership grew from an initial n = 37 (July 2019) to n = 322 individuals [May 2020; [Figure 4]a. Each of these in-person meetings solicited collaborative input from stakeholders toward execution of concrete regulatory science deliverables. [Figure 4]a also includes the number of participants and frequency of steering committee web meetings. By July 2019, it became clear that various stakeholders worked on or had interest in distinct topics that the Alliance subsequently organized into 8 working groups by autumn 2019 [Figure 4]b. These group topics are intended to align stakeholders with subject-matter expertise and interest. Clearly, some functional requirements are relevant for multiple groups. However, we hope to minimize such redundancies by providing clear documentation of projects through appropriate project management and frequent content updates. The names of the founding and current working group leaders are provided in [Figure 4]b. One example of a regulatory science deliverable is also provided per group [Figure 4]b. For further updates or details on the various topics, please visit the Alliance website[8] or to become a member and get involved.
Figure 4: Roadmap and working groups. (a) Roadmap of in-person events (status May 2020). In addition to the date, the roadmap shows hosting organization, key developments, and location of the meetings. The graph shows the membership number over time along with the number and frequency of the steering committee meetings as well as the high-throughput truthing working group. (b) The Alliance proposed to tackle regulatory science deliverables in digital pathology by splitting up the topic into eight distinct working groups. Each workgroup is provided with the steering committee member (s) and at least one key regulatory science deliverable. The steering committee is also responsible for minimizing redundancy between the workgroups. AI: Artificial intelligence; DPA: Digital Pathology Association; FDA: Food and Drug Administration; HTT: High-throughput truthing (an independent workgroup); MDIC: Medical Device Innovation Consortium; ML: Machine learning; USCAP: USCAP stands for United States and Canadian Academy of Pathology

Click here to view

   The Alliance facilitates Regulatory Submissions Top

As a first key regulatory science deliverable, in late 2019, members of the Alliance submitted an MDDT proposal to the FDA for review (HTT project described above). The experience gained through this submission will create a starting point and testing ground for the proposed approach of the Alliance. In contrast to the largely confidential submission owned by the submitting entity (typically represented through a consulting firm and/or a regulatory affairs division), gaining and sharing the submission experience may inform subsequent submissions, and Alliance members can draw from the experience of these submissions. This particular concept is new to digital pathology. Similarly, we consider several precompetitive submission programs by the FDA[78],[79] a paradigm shift that enables different ways to engage with regulatory entities. Importantly, the Alliance intends to create a repository of submission documents as a resource to bolster subsequent submissions with the collective experience of previous submitters. We propose that the field, and in particular patients,[80] will ultimately benefit from sharing the experiences of Alliance members who have submitted to regulatory agencies.

   Conclusion Top

In the current environment of sparse and dispersed regulatory guidance for digital pathology and AI/ML, with siloed pursuits by diverse stakeholders, the Alliance saw an opportunity to establish an important missing element: a precompetitive regulatory science collaboration. We believe that for patients to benefit from highly complex new technologies, benefit and risk assessments are essential.[81],[82] The Alliance helps tackle this daunting task (i. e., benefit and risk assessment for digital pathology and AI/ML) through regulatory sciences with the hope of successful clinical integration and improved patient care. That said, there are numerous issues that we need to address. For example, we want to investigate and develop protocols and definitions for continuous performance assessments of continuously learning ML algorithms. Similarly, approaching financial sustainability will require clear demonstration of clinical utility. However, the fact that numerous unanswered questions persist represents an opportunity for other agencies, regulatory entities, professional groups, and collaborative movements (like the Alliance) to step up and drive developments toward comprehensive risk and safety assessments. It is important to emphasize the crucial importance of funding for regulatory and implementation science projects, in particular those that aim to inform technically appropriate and efficient science-based regulatory decision-making processes. Such funding is needed to advance cutting-edge innovations into clinical practice. In summary, the Alliance aims to advance the field of digital pathology and we hope that synergistic efforts between various stakeholders and regulatory scientists will ultimately speed the improvement of patient care. This begs the question: Who, if not us?


The Alliance is supported by the Medical Device Innovation Consortium (Arlington, VA), the Digital Pathology Association and the Digital Pathology Association Foundation (Carmel, IN), and the Center for Integrated Diagnostics, Department of Pathology, Massachusetts General Hospital/Harvard Medical School (Boston, MA). This work is also in part supported by NIH (RO1 CA225655) to J.K.L, and the content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Health or any other organization.

Financial support and sponsorship


Conflicts of interest

There are no conflicts of interest.

   References Top

Weinberg AM. Science and trans-science. Science 1972;177:211.  Back to cited text no. 1
FDA. Advancing Regulatory Science; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 2
Abels E, Pantanowitz L. Current state of the regulatory trajectory for whole slide imaging devices in the USA. J Pathol Inform 2017;8:23.  Back to cited text no. 3
[PUBMED]  [Full text]  
Spindler P, Bach KF, Schmiegelow M, Bedlington N, Eichler HG. Innovation of medical products: the evolution of regulatory science, research, and education. Ther Innov Regul Sci 2016;50:44-8.  Back to cited text no. 4
FDA. Advancing Regulatory Science for Public Health; 2010. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 5
FDA. Regulatory Science in FDA's Center for Devices and Radiological Health: A Vital Framework for Protecting and Promoting Public Health; 2018.  Back to cited text no. 6
FDA. Collaborative Communities Toolkit; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 7
MGH. MDIC DF. The Alliance for Digital Pathology 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 8
Abels E, Pantanowitz L, Aeffner F, Zarella MD, van der Laak J, Bui MM, et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: A white paper from the Digital Pathology Association. J Pathol 2019;249:286-94.  Back to cited text no. 9
Aeffner F, Zarella MD, Buchbinder N, Bui MM, Goodman MR, Hartman DJ, et al. Introduction to digital image analysis in whole-slide imaging: A white paper from the digital pathology association. J Pathol Inform 2019;10:9.  Back to cited text no. 10
[PUBMED]  [Full text]  
Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A. Artificial intelligence in digital pathology New tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019;16:703-15.  Back to cited text no. 11
García-Rojo M. International Clinical guidelines for the adoption of digital pathology: A review of technical aspects. Pathobiology 2016;83:99-109.  Back to cited text no. 12
Guo H, Birsa J, Farahani N, Hartman DJ, Piccoli A, O'Leary M, et al. Digital pathology and anatomic pathology laboratory information system integration to support digital pathology sign-out. J Pathol Inform 2016;7:23.  Back to cited text no. 13
[PUBMED]  [Full text]  
Hosseini MS, Brawley-Hayes JA, Zhang Y, Chan L, Plataniotis K, Damaskinos S. Focus quality assessment of high-throughput whole slide imaging in digital pathology. IEEE Trans Med Imaging 2020;39:62-74.  Back to cited text no. 14
Lange H. Digital Pathology: A regulatory overview. Lab Med 2011;42:587-91.  Back to cited text no. 15
Mukhopadhyay S, Feldman MD, Abels E, Ashfaq R, Beltaifa S, Cacciabeve NG, et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: A multicenter blinded randomized noninferiority study of 1992 cases (pivotal study). Am J Surg Pathol 2018;42:39-52.  Back to cited text no. 16
Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol 2019;20:e253-61.  Back to cited text no. 17
Williams BJ, Bottoms D, Clark D, Treanor D. Future-proofing pathology part 2: Building a business case for digital pathology. J Clin Pathol 2019;72:198-205.  Back to cited text no. 18
Azar JC, Busch C, Carlbom IB. Histological stain evaluation for machine learning applications. J Pathol Inform 2013;4:S11.  Back to cited text no. 19
[PUBMED]  [Full text]  
Komura D, Ishikawa S. Machine learning methods for histopathological image analysis. Comput Struct Biotechnol J 2018;16:34-42.  Back to cited text no. 20
Ström P, Kartasalo K, Olsson H, Saolorzano L, Delahunt B, Berney DM, et al. Pathologist-level grading of prostate biopsies with artificial intelligence. arXiv preprint arXiv:1907.01368 (2019).  Back to cited text no. 21
Rivenson Y, Liu T, Wei Z, Zhang Y, de Haan K, Ozcan A. PhaseStain: The digital staining of label-free quantitative phase microscopy images using deep learning. Light Sci Appl 2019;8:23.  Back to cited text no. 22
Alba E. Optimization techniques for solving complex problems. In: Hoboken NJ, editor. Wiley: Telecom; 2009. p. 476.  Back to cited text no. 23
Frensch PA, Funke J. Complex problem solving: The European perspective. In: Hoboken NJ, editor. L. Erlbaum Associates: Psychology Press; 1 edition; 1995. p. 340.  Back to cited text no. 24
Malakooti B. Operations and Production Systems with Multiple Objectives. Hoboken, New Jersey: John Wiley & Sons Inc.; 2014. p. 1075.  Back to cited text no. 25
Lyles MA, Marjorie A, Mitroff II. Organizational problem formulation: An empirical study. Administrative Sci Quarterly 1980;25:102-19.  Back to cited text no. 26
Cucoranu IC, Parwani AV, Vepa S, Weinstein RS, Pantanowitz L. Digital pathology: A systematic evaluation of the patent landscape. J Pathol Inform 2014;5:16.  Back to cited text no. 27
[PUBMED]  [Full text]  
Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 2018;24:1559-67.  Back to cited text no. 28
Campanella G, Hanna MG, Geneslaw L, Miraflor A, Werneck Krauss Silva V, Busam KJ, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 2019;25:1301-9.  Back to cited text no. 29
Campanella G, Rajanna AR, Corsale L, Schüffler PJ, Yagi Y, Fuchs TJ. Towards machine learned quality control: A benchmark for sharpness quantification in digital pathology. Comput Med Imaging Graph 2018;65:142-51.  Back to cited text no. 30
FDA. 510(k) Premarket Notification Aperio AT2 DX System; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 31
FDA. FDA Allows Marketing ofFirst Whole Slide Imaging System for Digital Pathology; 2017. first-whole-slide-imaging-system- digital-pathology. [Last accessed on 2020 Jul 07].  Back to cited text no. 32
Hamilton PW, Bankhead P, Wang Y, Hutchinson R, Kieran D, McArt DG, et al. Digital pathology and image analysis in tissue biomarker research. Methods 2014;70:59-73.  Back to cited text no. 33
Ho J, Ahlers SM, Stratman C, Aridor O, Pantanowitz L, Fine JL, et al. Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization. J Pathol Inform 2014;5:33.  Back to cited text no. 34
Isaacs M, Lennerz JK, Yates S, Clermont W, Rossi J, Pfeifer JD. Implementation of whole slide imaging in surgical pathology: A value added approach. J Pathol Inform 2011;2:39.  Back to cited text no. 35
[PUBMED]  [Full text]  
Mills AM, Gradecki SE, Horton BJ, Blackwell R, Moskaluk CA, Mandell JW, et al. Diagnostic efficiency in digital pathology: A comparison of optical versus digital assessment in 510 surgical pathology cases. Am J Surg Pathol 2018;42:53-9.  Back to cited text no. 36
Pantanowitz L, Dickinson K, Evans AJ, Hassell LA, Henricks WH, Lennerz JK, et al. American Telemedicine Association clinical guidelines for telepathology. J Pathol Inform 2014;5:39.  Back to cited text no. 37
[PUBMED]  [Full text]  
Williams BJ, Bottoms D, Treanor D. Future-proofing pathology: The case for clinical adoption of digital pathology. J Clin Pathol 2017;70:1010-8.  Back to cited text no. 38
Baidoshvili A, Bucur A, van Leeuwen J, van der Laak J, Kluin P, van Diest PJ. Evaluating the benefits of digital pathology implementation: Time savings in laboratory logistics. Histopathology 2018;73:784-94.  Back to cited text no. 39
Fraggetta F, Garozzo S, Zannoni GF, Pantanowitz L, Rossi ED. Routine Digital Pathology Workflow: The Catania Experience. J Pathol Inform 2017;8:51.  Back to cited text no. 40
[PUBMED]  [Full text]  
Hanna MG, Reuter VE, Hameed MR, Tan LK, Chiang S, Sigel C, et al. Whole slide imaging equivalency and efficiency study: Experience at a large academic center. Mod Pathol 2019;32:916-28.  Back to cited text no. 41
Hanna MG, Reuter VE, Samboy J, England C, Corsale L, Fine SW, et al. Implementation of digital pathology offers clinical and operational increase in efficiency and cost savings. Arch Pathol Lab Med 2019;143:1545-55.  Back to cited text no. 42
Sirintrapun SJ, Lopez AM. Telemedicine in cancer care. Am Soc Clin Oncol Educ Book 2018;38:540-5.  Back to cited text no. 43
FDA. Laboratory Developed Tests. 2018;  Back to cited text no. 44
Schreier J, Feeney R, Keeling P. Diagnostics reform and harmonization of clinical laboratory testing. J Mol Diagn 2019;21:737-45.  Back to cited text no. 45
Zehnbauer B. Integration standardization and diagnostics oversight of laboratory testing. J Mol Diagn 2019;21:735-6.  Back to cited text no. 46
Halling KC, Schrijver I, Persons DL. Test verification and validation for molecular diagnostic assays. Arch Pathol Lab Med 2012;136:11-3.  Back to cited text no. 47
Clinical Laboratory Improvement Amendments of 1988. USA: Public Law; 1988. p. 100-578.  Back to cited text no. 48
CAP. Digital Pathology Resource Guide; 2017. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 49
Cucoranu IC, Parwani AV, Pantanowitz L. Digital whole slide imaging in cytology. Arch Pathol Lab Med 2014;138:300.  Back to cited text no. 50
Zarella MD, Bowman D, Aeffner F, Farahani N, Xthona A, Absar SF, et al. A practical guide to whole slide imaging: A white paper from the digital pathology association. Arch Pathol Lab Med 2019;143:222-34.  Back to cited text no. 51
Hartman DJ, Ahmad F, Ferris RL, Rimm DL, Pantanowitz L. Utility of CD8 score by automated quantitative image analysis in head and neck squamous cell carcinoma. Oral Oncol 2018;86:278-87.  Back to cited text no. 52
Rashidi HH, Tran NK, Betts EV, Howell LP, Green R. Artificial intelligence and machine learning in pathology: The present landscape of supervised methods. Acad Pathol 2019;6:2374289519873088.  Back to cited text no. 53
Herrmann MD, Clunie DA, Fedorov A, Doyle SW, Pieper S, Klepeis V, et al. Implementing the DICOM standard for digital pathology. J Pathol Inform 2018;9:37.  Back to cited text no. 54
[PUBMED]  [Full text]  
D'Angelo R, Weiss R, Wolfe D, Chinnam R, Murat A, Gluesing J, et al. Facing the inevitable: being prepared for regulatory requirements for laboratory developed tests. Am J Clin Pathol 2018;149:484-98.  Back to cited text no. 55
Genzen JR. Regulation of laboratory-developed tests. Am J Clin Pathol 2019;152:122-31.  Back to cited text no. 56
Genzen JR, Mohlman JS, Lynch JL, Squires MW, Weiss RL. Laboratory-developed tests: A legislative and regulatory review. Clin Chem 2017;63:1575-84.  Back to cited text no. 57
Kim AS, Bartley AN, Bridge JA, Kamel-Reid S, Lazar AJ, Lindeman NI, et al. Comparison of laboratory-developed tests and FDA-approved assays for BRAF, EGFR, and KRAS Testing. JAMA Oncol 2018;4:838-41.  Back to cited text no. 58
Walker Q. Designing a standard of proof: The case for professional standards in next-generation sequencing laboratory-developed tests. J Law Biosci 2017;4:216-26.  Back to cited text no. 59
Khullar D, Ohn JA, Trusheim M, Bach PB. Understanding the Rewards of Successful Drug Development-Thinking Inside the Box. N Engl J Med 2020;382:473-80.  Back to cited text no. 60
Sim I. Mobile Devices and Health. N Engl J Med 2019;381:956-68.  Back to cited text no. 61
Clunie D, Hosseinzadeh D, Wintell M, De Mena D, Lajara N, Garcia-Rojo M, et al. Digital imaging and communications in medicine whole slide imaging connectathon at digital pathology association pathology visions 2017. J Pathol Inform 2018;9:6.  Back to cited text no. 62
Research GV. Digital Pathology Market Size, Share & Trends Analysis Report By Application (Drug Discovery & Development, By Product, Academic Research, Diagnosis), By End Use, And Segment Forecasts, 2019 – 2026; 2019. Available from:  Back to cited text no. 63
MDIC. Press Release: Patient Preference Collaboration; 2017.  Back to cited text no. 64
MDIC. Framework for Developing Clinical Evidence for Regulatory and Coverage Assessments inIn vitro Diagnostics (IVDs); 2019. Available from:  Back to cited text no. 65
NEST. National Evaluation System for Health Technology Coordinating Center (NESTcc); 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 66
PCORI. Patient-Centered Research for Standards of Outcomes in Diagnostic Tests (PROD); 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 67
PCORI. The Role of Breast Density in Decision Making for Breast Cancer Screening and Diagnostic Testing-The BCSC-ADVANCE Study; 2016. Available from: -and-diagnostic. [Last accessed on 2020 Jul 07].  Back to cited text no. 68
FOCR. Tumor Mutational Burden (TMB); 2018. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 69
FOCR. ctDNA for Monitoring Treatment Response (ctMoniTR) Project; 2019.  Back to cited text no. 70
Colling R, Pitman H, Oien K, Rajpoot N, Macklin P, CM-Path AI in Histopathology Working Group, et al. Artificial intelligence in digital pathology: A roadmap to routine use in clinical practice. J Pathol 2019;249:143-50.  Back to cited text no. 71
Turner OC, Aeffner F, Bangari DS, High W, Knight B, Forest T, et al. Society of toxicologic pathology digital pathology and image analysis special interest group article*: Opinion on the application of artificial intelligence and machine learning to digital toxicologic pathology. Toxicol Pathol 2020;48:277-94.  Back to cited text no. 72
FDA. Real-World Evidence; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 73
Research FoC. Pilot Project 2.0: Establishing the Utility of Real-World Endpoints; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 74
DPA. A Brief Explanation of the Differences Between the DPA and the Alliance; 2019.  Back to cited text no. 75
Denkert C, von Minckwitz G, Brase JC, Sinn BV, Gade S, Kronenwett R, et al. Tumor-infiltrating lymphocytes and response to neoadjuvant chemotherapy with or without carboplatin in human epidermal growth factor receptor 2-positive and triple-negative primary breast cancers. J Clin Oncol 2015;33:983-91.  Back to cited text no. 76
TILS. International Immuno-Oncology Biomarker Working Group on Breast Cancer; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 77
FDA. Medical Device Development Tools (MDDT); 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 78
Myers KJ. Mock Submissions to FDA/CDRH: History and Lessons Learned; April, 2019. Available from:  Back to cited text no. 79
FDA. Patient Engagement in the Design and Conduct of Medical Device Clinical Investigations. 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 80
FDA. Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions; 2018. Available from: availability-compliance-and. [Last accessed on 2020 Jul 07].  Back to cited text no. 81
FDA, CDRH. Digital Health; 2020. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 82
Wicklung E. FDA Cracks Down on DIY mHealth Platforms for Diabetes Care Management. mHealth Intelligence; 2019. Available from: cracks-down-on-diy-mhealth-platforms-for-diabetes-care-management. [Last accessed on 2020 Jul 07].  Back to cited text no. 84
FDA. FDA Warns People with Diabetes and Health Care Providers Against the Use of Devices for Diabetes Management Not Authorized for Sale in the United States: FDA Safety Communication; 2019. Available from: -health-care-providers-against-use-devices-diabetes- management-not. [Last accessed on 2020 Jul 07].  Back to cited text no. 85
Hensley S. 23andMe Bows To FDA's Demands, Drops Health Claims. NPR 2013. Available from: drops-health-claims. [Last accessed on 2020 Jul 07].  Back to cited text no. 86
Kelly S. Software Workarounds put Patient data at Risk: OIG. Medtechdive; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 87
Team I. The Future Of Voice AI In Patient Care. Forbes; 2019. Available from:  Back to cited text no. 88
Eddy N. Overload often lead to staff workarounds. Healthcare IT News; 2019. Available from: [Last accessed on 2020 Jul 07].  Back to cited text no. 89


  [Figure 1], [Figure 2], [Figure 3], [Figure 4]

  [Table 1]




   Browse articles
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

  In this article
    Toward an Operat...
    The Multifaceted...
    Regulatory Scien...
    From Key Mission...
    Who is the Allia...
    Meetings, Growth...
    The Alliancefaci...
    Toward an Operat...
    The Multifaceted...
    Regulatory Scien...
    From Key Mission...
    Who is the Allia...
    Meetings, Growth...
    The Alliance fac...
    Article Figures
    Article Tables

 Article Access Statistics
    PDF Downloaded587    
    Comments [Add]    

Recommend this journal