|J Pathol Inform 2021,
Artificial intelligence in pathology: From prototype to product
André Homeyer1, Johannes Lotz1, Lars Ole Schwen1, Nick Weiss1, Daniel Romberg1, Henning Höfener1, Norman Zerbe2, Peter Hufnagl2
1 Fraunhofer MEVIS, Bremen, Germany
2 Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Institute of Pathology; HTW University of Applied Sciences Berlin, Berlin, Germany
|Date of Submission||29-Sep-2020|
|Date of Decision||28-Dec-2020|
|Date of Acceptance||18-Jan-2021|
|Date of Web Publication||22-Mar-2021|
Dr. André Homeyer
Fraunhofer MEVIS, Am Fallturm 1, 28359 Bremen
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Modern image analysis techniques based on artificial intelligence (AI) have great potential to improve the quality and efficiency of diagnostic procedures in pathology and to detect novel biomarkers. Despite thousands of published research papers on applications of AI in pathology, hardly any research implementations have matured into commercial products for routine use. Bringing an AI solution for pathology to market poses significant technological, business, and regulatory challenges. In this paper, we provide a comprehensive overview and advice on how to meet these challenges. We outline how research prototypes can be turned into a product-ready state and integrated into the IT infrastructure of clinical laboratories. We also discuss business models for profitable AI solutions and reimbursement options for computer assistance in pathology. Moreover, we explain how to obtain regulatory approval so that AI solutions can be launched as in vitro diagnostic medical devices. Thus, this paper offers computer scientists, software companies, and pathologists a road map for transforming prototypes of AI solutions into commercial products.
Keywords: Artificial intelligence, business model, image analysis, integration, pathology, regulatory approval, reimbursement, technology readiness
|How to cite this article:|
Homeyer A, Lotz J, Schwen LO, Weiss N, Romberg D, Höfener H, Zerbe N, Hufnagl P. Artificial intelligence in pathology: From prototype to product. J Pathol Inform 2021;12:13
|How to cite this URL:|
Homeyer A, Lotz J, Schwen LO, Weiss N, Romberg D, Höfener H, Zerbe N, Hufnagl P. Artificial intelligence in pathology: From prototype to product. J Pathol Inform [serial online] 2021 [cited 2021 Jul 31];12:13. Available from: https://www.jpathinformatics.org/text.asp?2021/12/1/13/311693
∗André Homeyer, Johannes Lotz, Lars Ole Schwen, and Nick Weiss contributed equally.
| Introduction|| |
Artificial intelligence (AI) has the potential to revolutionize pathology.,, AI refers to the application of modern machine learning techniques to digital tissue images in order to detect, quantify, or characterize specific cell or tissue structures. By automating time-consuming diagnostic tasks, AI can greatly reduce the workload and help to remedy the serious shortage of pathologists. At the same time, AI can make analyses more sensitive and reproducible and it can capture novel biomarkers from tissue morphology for precision medicine. In a survey from the year 2019 of 487 pathologists from 54 countries, a great majority looked forward to using AI as a diagnostic tool.
A wide variety of diagnostic tasks in pathology benefits from AI support. Even general tasks, such as counting nuclei or classifying tumor tissue, usually must be solved in a disease- and tissue-specific manner. A PubMed search for the terms “artificial intelligence” or “machine learning” in conjunction with “pathology” resulted in more than 5000 publications, half of which were published since 2018 (as of September 2020). Several overview papers were published on how AI techniques can be applied in pathology.,,,, Recent reviews discuss AI applications in pathology and specifically in breast pathology., Another much-studied application of AI is the extraction of novel immuno-oncology biomarkers from pathology images.
Despite the extensive research developments, very few AI solutions have become commercial products for routine use. As a result, a large part of the potential of AI remains untapped. Various technical, business, and regulatory challenges must be overcome in order to commercialize AI solutions in pathology. Similar challenges also arise for AI solutions in other medical domains, such as radiology,, and ophthalmology. Research prototypes must be further developed and integrated into the IT infrastructure of clinical laboratories before they are usable in routine pathology workflows. Commercial success requires a profitable business model and the ability for pathologists to be reimbursed for using the product. AI solutions are medical devices and therefore require regulatory approval before they may be sold as a product.
Many publications about AI in pathology point to specific challenges, such as obtaining regulatory approval or reimbursement.,,,, A recent paper gives a detailed outline of the regulatory steps for developing clinical AI software. Still, it remains a major effort for AI solution developers to gather the wealth of knowledge required to meet the challenges of commercialization. In the present paper, we aim to provide researchers, software companies, and clinicians with a comprehensive overview on how to transform prototypes of AI solutions into commercial products. To this end, we provide basic knowledge and advice on all the challenges mentioned above.
| Software Maturity|| |
The main objective in developing research prototypes is to demonstrate the feasibility of a task. Once feasibility has been verified, research prototypes still need to mature in various ways to meet the requirements for routine use: The software must be of sufficient quality to be stable and maintainable. The solution must also run efficiently enough for the laboratory workflow and reliably deliver accurate results.
Research prototypes of AI solutions are often written quickly without considering code quality. This makes such prototypes expensive to maintain and to extend by new features, i.e., liable to “technical debt”. Moreover, it makes prototypes prone to runtime errors and vulnerable to security holes. For these reasons, software quality often needs to be improved before a research prototype is ready for commercialization. Software quality is also important for obtaining regulatory approval. While regulatory approval does not explicitly prescribe a certain degree of software quality, one must adhere to a quality management system (QMS) throughout the entire development process, which should lead to high quality.
Various measures can be taken to improve software quality. Research code often has grown gradually and lacks structure. In this case, it helps to improve its architecture, e.g., by encapsulating functional units in a modular way and by implementing object-oriented design patterns. For achieving stable and properly functioning code, extensive testing is indispensable. Testing should be carried out automatically, both by low-level unit tests and high-level functional and integration tests. Ideally, source code should be developed in a test-driven manner right from the beginning.
Research prototypes of AI solutions for pathology are usually optimized for accuracy, often at the expense of computing efficiency. Such prototypes must be further improved to be fast enough (while still accurate enough) for laboratory practice. The first step in optimizing efficiency should always be to identify the specific performance bottlenecks. In addition, there are several general strategies to make AI solutions for pathology more efficient.
One strategy is to divide long-running operations into several smaller operations that can be executed in parallel. Tensor processing operations, which are performed in large quantities in deep learning-based algorithms, are parallelized automatically by all major machine learning libraries. Similarly, slide images can be divided into smaller “tiles” and processed in parallel. Another common strategy is to apply an algorithm to a lower resolution image which is substantially faster to process, if this does not affect accuracy in a relevant way.
Research prototypes of AI solutions have only been trained and tested on limited datasets provided by a clinical project partner or in the context of a challenge. Such datasets hardly cover the real-world variability of tissue images. The variability of tissue images is enormous, as their appearance is influenced by the depicted tissue types, pathologies, and the processing of the tissue section, including its cutting, staining, and scanning. For this reason, research prototypes often need to be trained further on larger and more representative datasets before becoming sufficiently reliable for routine use.
There are no simple criteria for deciding whether a particular dataset is representative. In general, datasets should be collected in a consistent and well-curated form from several laboratories using individual preparation and scanning procedures and covering different ethnic groups and disease subtypes.,,
The need for collecting large datasets can be somewhat reduced by incorporating synthetic data in the training process. The most popular technique is data augmentation, in which several transformations are derived from the original training data according to some variability model. Besides standard transformations such as rotations and changes in contrast, more advanced data augmentation techniques can be employed, e.g., to simulate staining variability., Another recent approach is using Generative Adversarial Networks to automatically create realistic synthetic histology images.,
Technology readiness levels
In addition to software maturity, there are further technical requirements to make AI solutions in pathology ready for routine use. These include integrability into the laboratory IT infrastructure and compliance with regulatory technical requirements, as described in the next sections.
Technical readiness is commonly classified in technology readiness levels (TRLs) on a spectrum from an initial idea to a solution for productive use. As such, TRLs are a practical measure for both internal product management and communication with investors. Originating from aerospace engineering, TRLs have been used in different domains with slightly varying definitions.,,, We have adapted the original definition of TLRs to illustrate the steps required to turn a prototype AI solution in pathology into a product [Figure 1].
|Figure 1: Technology readiness levels for artificial intelligence solutions in digital pathology|
Click here to view
A typical research prototype corresponds to TRL 3 if it was evaluated in a peer-reviewed publication, or TRL 4 if it has proven itself in a community challenge., Product development is an iterative process requiring input and feedback from the intended users. TRL 5 provides initial information for defining efficiency and reliability requirements. Fulfilling these requirements is necessary to reach TRL 7, i.e., a clinical prototype with demonstrated utility in an operational environment. The clinical prototype is the basis for obtaining regulatory approval and ultimately achieving product status on TRL 9.
| Integration|| |
Research prototypes of AI solutions in pathology are usually standalone tools with their own interfaces for execution, data input, and data output. To be usable in diagnostic routine, the tools must be integrated into the laboratory IT infrastructure and made interoperable with software from other manufacturers.
Laboratory IT infrastructure
Three pieces of software are of particular importance for using AI solutions in pathology [Figure 2]. The central component of any laboratory IT infrastructure is the Pathology Laboratory Information System (PLIS). Its core functionality is case and sample management, especially storing all received and generated data on specimens and tracking specimen-related data during histological processing. Many PLISs also support work planning of laboratory personnel and coding and billing of diagnostic procedures.
|Figure 2: Communication paths between an artificial intelligence solution and the main IT components in a digital pathology lab|
Click here to view
In digital pathology laboratories, another important piece of software is the image archive, which manages and provides access to slide images and their associated metadata. Manufacturers of slide scanner devices offer special slide server software optimized for their own image formats. In addition, there are Picture Archiving and Communications Systems (PACSs) that manage and provide access to medical image data in a standardized way. Such PACSs were originally developed for radiology images, but are now also available for pathology images. Health-care institutions are increasingly using vendor-neutral archives (VNAs) that integrate various types of medical images, including pathology images. Such VNAs greatly facilitate sharing image data between departments and reduce maintenance costs by consolidating multiple storage systems into a single central solution. A recent publication provides guidance on the use of VNAs in pathology.
The digital pathology workstation is the software with which the pathologist interacts most when assessing a case. Its user interface is centered around a virtual microscope viewer providing means for exploring and interacting with a slide image, e.g., to make measurements or annotations. For this purpose, the workstation is closely linked to the image archive. Many workstations also provide functionality for accessing additional patient information or submitting diagnostic reports by communicating with the PLIS. Slide scanner manufacturers typically offer special workstation software for their devices. However, there is also an increasing number of vendor-neutral or open-source workstation software.,
Communication with PLISs works according to general clinical messaging standards that define how to encode patient data, specimen data, and diagnostic reports. The most widespread standard Health Level Seven International (HL7) V2 was developed by the HL7 organization in the 1980s. The latest standard called HL7 Fast Healthcare Interoperable Resources (FHIR) is based on current Internet technologies and easier to use. Image archive systems offered by slide scanner manufacturers usually exhibit proprietary interfaces for accessing image data. In contrast, PACSs and VNA systems enable uniform image access through the Digital Imaging and Communications in Medicine (DICOM) standard. Originating from radiology, the DICOM standard is also becoming increasingly popular in pathology.
Integration of artificial intelligence
AI solutions can be integrated into the laboratory IT infrastructure either as standalone tools or workstation extensions.,,, Standalone tools provide their own user interface for triggering an analysis, for selecting input parameters, such as regions-of-interest or algorithm settings, and for presenting analysis results. With workstation extensions, all these functions are performed through the user interface of their host software. Workstation extensions must implement a predefined programming interface through which they are triggered and input and output data are exchanged.
Workstation extensions, on the other hand, have the major advantage of workflow efficiency. They can be used without leaving the familiar workstation environment and without having to become acquainted with new tools. Another advantage of workstation extensions is lower development costs. Standalone tools must be interoperable with PLIS, image archive, and workstation software in order to retrieve input data or submit results. Given the different types of interfaces, this can be a major effort. For workstation extensions, this functionality is already provided by their host and does not need to be implemented again.
Because of their respective advantages and disadvantages, standalone tools are mainly offered by well-established manufacturers with a comprehensive portfolio of solutions for different applications., For new companies intending to turn a prototype into a product, it can be easier to offer their solution as a workstation extension.
There is no standard for integrating AI solutions into workstation software, but each system offers its own specific interface. The interface definitions of commercial workstation software are often not public and only accessible through partnerships with the respective manufacturers. While some workstations integrate AI solutions as dynamically-linked software components, most modern systems integrate them as Docker containers or RESTful Web services., The latter technologies have the advantage of allowing solutions to be implemented in any programming language. They also enable distributed operation of the solution on optimized hardware (e.g., GPUs), which is particularly important when applying AI technologies to large slide images.
| Business Model|| |
Successful development and sale of AI solutions for pathology require a sustainable business model. A business model is defined as “the rationale of how an organization creates, delivers, and captures value”. At the heart of every business model is a value proposition for customers. Around this, a business model covers the conditions required to create that value, how to deliver that value to customers, and how to generate revenue.
Business model canvas
A popular approach to developing a business model are canvases, i.e., large poster-like forms summarizing different aspects of the business model. Following the popular business model canvas template, we discuss the aspects for AI solutions in pathology [Figure 3]. We here focus on technology-driven start-ups aiming to bring a technology into the clinical market rather than exit-driven business models where the product is the start-up itself.
|Figure 3: Example business model canvas for a pathology artificial intelligence solution: It is recommended to construct the business model starting with the right side that focuses on the customer before addressing the left side that addresses the business itself. This figure is based on the Strategyzer business model canvas, licensed CC-BY-SA|
Click here to view
Customers decide on the purchase of a solution. They are not always identical to users, but their interests often overlap. There are two main customer segments for image analysis solutions in pathology: the first are pathology laboratories or hospitals who are also users of the solutions. The second are vendors of pathology workstations or device manufactures who integrate and distribute third-party solutions with their software—often as white label products under their own name.
The value proposition is what makes the customer pay money for a solution and can differ between customer segments. Assumptions about the value proposition need to be validated as early as possible, e.g., through customer relations or market research. For AI solutions in pathology, the principal value propositions are cost reduction, quality improvement, and innovation. AI solutions can reduce costs by assisting pathologists and enabling them to perform more work in less time. Pathologists can focus more on complicated cases by saving time on simple cases. Quality improvement means higher accuracy or sensitivity of diagnostic tests. Finally, AI solutions can compute innovative computational biomarkers that are prognostic for the course of a disease or the effectiveness of a therapy.
AI solutions can be distributed through direct or indirect sales channels. It is common practice for developers of AI solutions to partner with manufacturers of scanner devices or pathology software systems. The respective products are then either cross-sold directly by both manufacturers, or the AI solution is integrated into existing software and distributed indirectly through the respective manufacturer. Particularly for new companies, it can be attractive to enter business development deals with established manufacturers and to benefit from their market reach and marketing.
In recent years, app stores have become a common way of distributing software. App stores facilitate multiple aspects of a business model such as distribution, advertising, purchase processing, and customer feedback. Although there is not yet a vendor-neutral app store where developers can sell AI solutions for pathology, the first web-based directories have been launched.,
Building customer relationships is important to validate and adapt the value proposition according to the customer needs. Important points of contact between developers of AI solutions and customers are scientific congresses, the company website, and social media. Furthermore, webinars and user group meetings are commonly hosted in order to showcase new products or features and collect feedback in a systematic way. Especially in the early phase of a company, it can make sense to offer freemium software, or free trials, since testing the value proposition may be more valuable than first sales.
AI solutions for pathology can be licensed for a one-time payment, as part of a subscription service, or on a pay-per-use basis. Recurring payment models are becoming increasingly popular in the software market. Subscription services ensure a continuous revenue stream for the developer while entitling the customer to the latest updates. Pay-per-use minimizes the entry barrier for the customer, but also leads to an uncertain revenue stream for the developer.
The realized revenues depend strongly on the pricing strategy. In cost-based pricing, the costs of development and maintenance determine the price of the product, which may result in a price below the customer's willingness to pay. In competition-based pricing, the aim is to achieve the price of a competitor. Customer-based pricing estimates the value that the product brings to the customer, such as the cost savings from faster analysis or the reimbursement paid for computer assistance.
Key resources, activities, and partners
Key resources for creating an AI solution for pathology are a development team with experience in machine learning and professional software development, computing resources to develop software and train machine learning models, and perhaps most importantly, data. Since the quality of an AI solution highly depends on the quality and representativeness of the training data, a large number of tissue images and high-quality annotations must be obtained from several laboratories.
Key activities include demonstrating clinical utility and robustness, establishing a distribution channel, and obtaining regulatory approval. If the value proposition focuses on quality improvement instead of cost reduction, establishing reimbursement options is also an important activity.
Key partners are pathologists who assist in identifying opportunities for computer support, who acquire tissue images and annotations, and who provide feedback on the usefulness of a solution. Other important partners are manufacturers of scanner devices or pathology software systems who integrate and distribute AI solutions as part of their software platforms.
The main cost factors in the development of AI solutions for pathology are personnel and the procurement of sufficient amounts of training and testing data. Other significant costs arise from office operations, computing resources, fees for consulting pathologists, and marketing. In technology-driven business models, a large part of the value lies in the skills and knowledge of the employees. Hiring highly qualified employees is therefore often a higher priority than reducing personnel costs. Furthermore, in the early stages of a new business, winning more customers and growing the business is often a higher priority than cutting costs, provided that financing allows for this.
Iterative business models
A novel business model is based on many assumptions. To reduce the risk of failure, one should initially focus on the quick release of a minimum viable product (MVP). An MVP contains only essential features necessary to win first customers. This makes it possible to receive early feedback and validate the assumptions of the business model.
The need to obtain regulatory approval makes it very costly to develop AI solutions for routine use. A cheaper way to test the utility of a new solution and detect potential shortcomings is to provide an analysis service from a central laboratory as an intermediate business model. Other laboratories can send in physical tissue slides or digital tissue images and the central laboratory returns a report about the analysis results. The central laboratory model is already established for complex or proprietary immunohistochemical (IHC) tests that cannot be conducted at local laboratories.
The central laboratory model can be implemented by offering AI solutions as laboratory-developed tests (LDTs). With some limitations, LDTs may use any hardware or software components without requiring them to be certified medical devices. Instead, the laboratory as a whole must be accredited to work competently and the test must be validated in-house, which is usually less costly than the approval of a medical device. Once an AI solution has proven itself within the central laboratory model, the effort of certification as a medical device is worthwhile in order to reach a larger customer group.
| Regulation|| |
In most countries, medical products may only be distributed with regulatory approval. From a regulatory point of view, AI solutions for pathology are in vitro diagnostic medical devices (IVDMDs), as they evaluate images of body fluids or tissue outside the body for diagnostic or therapeutic purposes. We here describe different concepts relevant in the regulation of IVDMDs, focusing on the EU and the USA as two examples. In this context, we address specific challenges and current developments in the regulation of AI solutions.
The approval process for medical devices varies between legislations and some countries have separate regulations for IVDMDs., In the EU, IVDMDs were regulated by Directive 98/79/EC until 2017. After a transition phase until May 26, 2022, IVDMDs will be regulated by Regulation 2017/746. General medical devices, which are not IVDMDs, are regulated separately. In the USA, regulatory procedures for IVDMDs are defined in the Code of Federal Regulations, Title 21 and are the same as for other medical devices.
Agencies handling the regulatory process of IVDMDs are structured differently in different legislations: they can be part of the ministry of health (e.g., in China and Israel), a specialized agency (e.g., the U. S. Food and Drug Administration [FDA]), or the task can be delegated further (e. g., EU, India or Japan).,, One example for delegation are the Notified Bodies in the EU, which are independent organizations handling the actual approval process. Manufacturers are free to choose with which Notified Body in their country they want to work.
Medical devices are commonly classified according to their risk for the patient. In the EU, AI solutions in pathology are considered class C devices (on a scale from A to D with increasing risk) according to Annex VIII, 2.3 of. The FDA considers AI solutions in pathology class II medical devices (on a scale from I to III with increasing risk).,
Regulatory approval procedures
The approval of medical software usually requires compliance with quality standards in the development process and an assessment of the product [Figure 4].
|Figure 4: Overview of documents necessary for obtaining regulatory approval|
Click here to view
Some legislations have different pathways for approval depending on the novelty and the risk of the respective device. As an example, the FDA uses specific terminology for different types of what is generally denoted as regulatory approval. High-risk (class III) devices need to be “approved” by the FDA before entering the market in the USA, whereas class II devices are “cleared” after passing a less complex process. In case of positive evaluation via the “de novo” pathway (see below), devices are “granted.” In either case, the product may legally be marketed afterward (where “marketing” in the regulatory context denotes entering the market at one point in time, and pre-/postmarket refers to what needs to be done beforehand/afterward).
Having established a QMS is one requirement for regulatory approval (e. g., Annex IX of in the EU or Part 820 of in the USA). This is commonly demonstrated by compliance with ISO 13485 or similar national standards. Other relevant standards for AI solutions in pathology are ISO 14971 for risk management and IEC 62304 for software life cycle processes.
The assessment of a product requires generating and submitting extensive documentation:
- Technical documentation, a description of requirements, architecture, and test results;
- Performance evaluation, see below; and
- A plan for postmarket surveillance, proactive monitoring, and addressing incidences.
In the EU, these requirements are described in Annexes II, XIII, and III of and need to be submitted for the specific product.
For a device to be cleared by the FDA in the USA, the manufacturer must demonstrate that the device is substantially equivalent to another device with FDA clearance or approval. The manufacturer must submit a 510(k) premarket notification submission to the FDA, who clears the device if equivalence is demonstrated., Otherwise, the manufacturer must request a de novo classification and demonstrate the safety and efficacy of the device.
In the EU, the manufacturer may use the Conformité Européenne (CE) mark for the device after having successfully completed the approval process. Other agencies (e.g., the FDA) offer a database of approved devices on their website.
In addition to the establishment of a QMS, the proper functional performance of the product is also assessed in regulatory procedures. As an example, the performance evaluation required in the EU has three aspects (Annex XIII of):
- Scientific validity: showing that there is a valid association between software output and targeted clinical condition,
- Analytical performance: showing that the software produces accurate and reliable output, e.g., by comparison to a reference approach for analysis, and
- Clinical performance: showing that the output meets the intended purpose in a clinical context and for the target population.
The performance evaluation for FDA clearance is similar, and it may be less effort if equivalence to another product is shown.
The performance evaluation is generally conducted in three steps: (a) writing a plan for each part of the evaluation, (b) assessing the tool according to the plan, and (c) writing a report on the results. These tasks require extensive amounts of validation data to be acquired by the manufacturer.
Development of AI-based solutions and the final products typically use third-party components, e.g., machine learning libraries. Such components are off-the-shelf software (OTSS) and/or software of unknown/uncertain pedigree/provenance (SOUP) from a regulatory point of view. The manufacturer of the AI-based solution needs show that the OTSS or SOUP fulfills the requirements defined for the product at hand.
Requirements for national approval vary if a software has already been approved elsewhere. For instance, a CE mark or FDA clearance is sufficient for an IVDMD to enter the market in India and certain South American nations. Other countries, such as Israel, only require some additional administrative steps. In the USA, on the other hand, a completely new approval procedure is required, even if the devices were previously approved elsewhere.
Requirements for the QMS of the manufacturer may differ in details but are generally similar to those in the ISO 13485 standard. Similarly, parts of performance analyses can be re-used even if details in the requirements differ between legislations. There are, however, initiatives to harmonize medical device regulation internationally.,
Regulation of artificial intelligence solutions
Image analysis software for digital pathology has been developed for many years and some current solutions are already AI-based (e.g.). However, AI-based software still poses certain challenges for regulation, such as novelty for specific application, opaqueness of algorithms, representativeness of training data, and evolving algorithms.
Existing scientific literature on AI solutions for specific applications may be sparse to nonexistent or of uncertain quality. This makes a critical assessment of the scientific state of the art, which is necessary in the context of performance evaluation, difficult.
Training and validation of AI algorithms require data sufficiently representative for the target applications to avoid bias., Assessing whether data are representative, however, is subject to many assumptions. Moreover, algorithms must be robust with respect to small changes in datasets and corrupt input data.
The fact that AI algorithms adapt to new data makes them “uncharted territory for organizations that regulate medical devices.” If only the manufacturer trains an algorithm and each new or re-trained version remains static, the performance evaluation can be carried out in advance, which is easier to handle in the regulatory approval process. If, however, the algorithm is to be further trained by the user, it becomes dynamic after approval and a “moving target for regulators”. Moreover, algorithms using AI (e.g., neural networks) usually act as “black boxes” lacking transparency and explainability how analysis results are obtained.,
Regulatory authorities like the FDA are currently in the process of establishing clear guidelines and procedures for AI-based IVDMDs., As a result, almost no AI solutions have been approved for in vitro diagnostics to date. In March 2019, the FDA for the first time classified an AI solution for pathological cancer diagnostics as a “breakthrough device” to accelerate product development and regulatory review. This solution for the detection of prostate cancer was also the first deep-learning-based solution to receive European IVD approval.
Obtaining regulatory approval for IVDMDs is challenging already in one country, and even more so when scaling internationally. In addition, regulatory procedures for AI-based IVDMDs are currently being updated. To cope with this complexity, it can be necessary to seek help from specialized consultants and legal experts.
| Reimbursement|| |
The payer of diagnostic pathology services is either a health insurance entity or the patient. Pathologists claim reimbursement for their services from these payers. AI solutions for pathology cause investment, maintenance, and operating costs, e.g., for licensing, IT administration, or manual supervision. For AI solutions to be used in practice, it is thus essential that pathologists can be reimbursed for their costs.
Billing medical services is extremely complex. Reimbursement systems vary from country to country. Even within countries, the procedures and amounts of reimbursement often vary between health insurance entities, insurance plans, and between regions. This section provides basic knowledge about reimbursement for AI solutions in pathology, using the USA and Germany as examples, to facilitate the establishment of reimbursement opportunities for newly developed products.
Most people in the USA are enrolled in private health insurance. In addition, there are public health insurance programs like Medicare and Medicaid covering elderly or low-income people. In Germany, most people are enrolled in statutory health insurance funds. It is also possible to take out private health insurance with an insurance company.
Most reimbursement systems are based on a kind of fee schedule that lists the content and fees of the various medical services [Figure 5]. Diagnostic pathology services provided to Medicare patients in the USA are reimbursed according to the Medicare Physician Fee Schedule (MPFS). The MPFS is often taken as the basis for the fee schedules of private health insurance plans and its rates are an upper limit on the fees reimbursed by Medicaid. The fees for clinical diagnostic laboratory tests are not listed in the MPFS but in the separate Clinical Laboratory Fee Schedule (CLFS).
|Figure 5: Fee schedules are negotiated between different stakeholders and determine reimbursement for diagnoses in digital pathology. This ultimately influences the revenues of artificial intelligence solution developers in digital pathology|
Click here to view
Germany has two basic medical fee schedules. The “Einheitlicher Bewertungsmaßstab” (EBM) schedule applies to services billed to statutory health insurers and the “Gebührenordnung für Ärzte” (GOÄ) applies to billings to private health insurers. The EBM only covers services deemed necessary and economical. Patients with statutory health insurance can obtain services not listed in the EBM on a self-pay basis, which are then billed according to the GOÄ.
Most fee schedules in the USA use the Current Procedural Terminology (CPT) code system for identifying medical services. Medicare or Medicaid use an extension to the CPT called Healthcare Common Procedure Coding System (HCPCS). The EBM and GOÄ fee schedules in Germany use individual code systems for identifying services.
Reimbursement of novel procedures
In the USA, the CPT system and the MPFS are updated annually. While the EBM in Germany is continuously being revised, the GOÄ was last updated in 1996 and therefore does not consider the latest medical progress. The GOÄ, however, gives physicians the opportunity to value novel medical services not listed in the schedule “analogously,” i.e., according to a listed service that is equivalent in type, cost, and time (see, § 6).
The national associations of pathologists play a key role in introducing new procedures in the fee schedules. In the USA, the College of American Pathologists (CAP) administers the Pathology Coding Caucus, which makes recommendations about the inclusion of new procedures into the CPT. Furthermore, the CAP advises on the fees for new diagnostic pathology services in the MPFS. In Germany, the Federal Association of Pathologists maintains a dedicated fees commission, which makes proposals for changes to the EBM or GOÄ and which publishes recommendations on the analog valuation of pathology services.
For a new diagnostic procedure to become reimbursable, its clinical utility must be demonstrated. The clinical utility of a procedure refers to its ability to improve patient outcomes, such as extensions of survival or reductions of morbidity. The clinical utility also refers to whether the benefits of a procedure are justified by its costs or effort. Diagnostic procedures can only indirectly improve patient outcomes by enabling better treatment decisions. Therefore, it can be helpful to develop a review framework to clarify the relationships between test results, clinical decisions, and patient outcomes. The clinical utility of diagnostic procedures is typically demonstrated by comparison with existing tests. In the USA, there is a program for the concurrent review of analytical and clinical performance for regulatory approval and clinical utility for Medicare coverage determination.
Reimbursement for computer assistance
Most AI solutions in pathology offer some kind of computer assistance based on automated image analysis. None of the fee schedules in the USA or Germany contain general listings for computer assistance in pathology or, in particular, for the use of AI. However, as explained in the following three examples, it is already possible to obtain reimbursement for certain diagnostic procedures involving computer assistance. The additional valuation of computer assistance differs significantly between procedures, between countries, and often also over time.
Screening for cervical cancer
Many countries offer cytological screening services to women to detect cervical cancer early. Computer assistance in the analysis of cytological samples can reduce screening errors and cause a significant increase in sensitivity.
Cervical cancer screening tests in the USA are reimbursable under the CLFS. For conventional cytology, screening with computer assistance was valued US$ 16 higher than manual screening in 2020 (HCPCS G0148-P3000). For liquid-based cytology, which offers better sample quality, the use of computer assistance was valued US$ 6 higher (HCPCS G0145-G0123). Both examples refer to location-guided screening, where the computer automatically identifies a certain number of points of interest which must then be reviewed manually. The German EBM and GOÄ do not distinguish between manual or computer-assisted CC or LBC and therefore do not allow for additional reimbursement of computer assistance.
DNA image cytometry
DNA image cytometry is sometimes used to identify or grade tumors based on irregularities in the DNA content of cells, e.g., as a supplement to Gleason grading in prostate cancer. With computer assistance, considerably more cells can be evaluated, making the analysis much more sensitive.
While the MPFS in the USA does not contain a listing for DNA image cytometry, this is one of the few procedures with computer assistance that can be billed in Germany. Conventional DNA image cytometry without computer assistance can be billed according to the EBM code 19330 at € 31.42. Since 2015, the procedure with computer assistance can be billed to the patient on a self-pay basis at a price of € 193.74. There is no dedicated code for the procedure, but it is valued analogously according to the GOÄ.
IHC biomarkers measure the expression of proteins relevant in tumorigenesis (e.g., ER/PR, PD-L1, Ki67, or HER2). They provide valuable information about the course of cancer or the likelihood of response to therapy. Computer assistance enables the automatic quantification of protein expression and thus makes the analysis of IHC biomarkers more accurate and reproducible.
In the USA, the evaluation of IHC biomarkers is reimbursable under the MPFS since 2004. Manual or computer-assisted assessment can be billed under CPT/HCPCS codes 88360 or 88361, respectively. The additional fees for computer assistance have steadily decreased over the years. While the fees for both codes differed by US$ 31.74 in 2009, the difference in 2020 was only US$ 1.80. The German fee schedules do not list any dedicated fees for the use of computer assistance in the evaluation of IHC biomarkers.
| Conclusions|| |
As described in the previous sections, a multitude of challenges must be overcome for turning prototypes of AI solutions in pathology into profitable products. These challenges can best be tackled in cooperation.
Cooperation is crucial
It is important to involve pathologists from an early stage of development. To obtain sufficient quantities of training and test data, partnerships must be established with pathologists and incentives should be provided for them to provide image data and required annotations. It is essential to enter into such partnerships with multiple laboratories in order to obtain a sample of interlaboratory variability. In order to develop a successful product, it is indispensable to take the pathologists' needs into account. Therefore, pathologists should be regularly provided with new prototypes and encouraged to test the prototypes extensively and provide systematic feedback. In view of the novelty and the high entry barrier to digital pathology, it is important to offer comprehensive support to pathologists, e.g., through user meetings and webinars.
Besides pathologists, it can be beneficial to collaborate with workstation manufacturers. The number of pathology laboratories working digitally is still small and there are no widely used distribution channels. By building strategic partnerships with workstation manufacturers who distribute AI solutions as part of their software, one can benefit from existing sales channels and customers.
Solution developers should also get involved with pathology associations. Cooperation with the national professional associations of pathologists is essential to establish reimbursement opportunities for new diagnostic procedures. In recent years, several organizations were founded specifically for the advancement of digital pathology, such as the Digital Pathology Association, the Alliance for Digital Pathology, and the European Society for Digital and Integrative Pathology. It can pay off to join these organizations in order to participate in shaping regulatory processes and technical standards.
There are further challenges in producing AI solutions that are beyond the scope of this paper. This concerns the many algorithmic challenges associated with AI development, which have already been addressed in numerous publications.,,,, Likewise, this concerns issues of usability and user experience. From a business perspective, a major challenge is setting up a company and financing the initial phase while the business model is not yet profitable. There is also extensive literature on this subject, e.g., on applying for public funding, or participating in incubator or accelerator programs.
One of the biggest barriers to the use of AI in pathology remains the inadequate IT infrastructure in most pathology laboratories. Only few laboratories have digital slide scanners at their disposal. Existing pathology software, such as PLISs, is often years old and lacks support for modern software technologies to integrate AI solutions (like web-based APIs or containerization). For the use of AI in pathology to become widespread, laboratories must be equipped with state-of-the-art digital pathology systems as standard.
Besides products for clinical-diagnostic purposes, which we focused on in this paper, AI solutions can also be turned into products for research purposes, e.g., to calculate tissue parameters for scientific studies. Research solutions have much lower requirements for product quality and robustness. Moreover, obtaining regulatory approval and reimbursement are irrelevant. Compared to the clinical market, however, the research market is much smaller.
Despite the challenges, there has never been a better time to bring AI solutions for pathology to market. Thanks to the latest AI technologies, automated image analysis has reached a level of robustness required for diagnostic use. Furthermore, digital pathology is finding its way into laboratory practice. Multiple vendors offer scanners and workstation software that are technologically mature and cleared for primary diagnosis. Finally, the potential of AI in pathology is widely recognized and multiple countries are funding initiatives to establish digitization and AI-based diagnosis in pathology.,,,
The authors would like to thank Sabrina Haase for fruitful discussions about regulatory affairs and Wolfgang Daniel Schmitt for advice on reimbursement.
Financial support and sponsorship
This work was funded by the German Federal Ministry for Economic Affairs and Energy via the EMPAIA project (grant numbers 01MK20002B, 01MK20002A).
Conflicts of interest
There are no conflicts of interest.
| References|| |
Niazi MK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol 2019;20:e253-61.
Parwani AV. Next generation diagnostic pathology: Use of digital pathology and artificial intelligence tools to augment a pathological diagnosis. Diagn Pathol 2019;14:138.
Robboy SJ, Weintraub S, Horvath AE, Jensen BW, Alexander CB, Fody EP, et al
. Pathologist workforce in the United States: I. Development of a predictive model to examine factors influencing supply. Arch Pathol Lab Med 2013;137:1723-32.
Sarwar S, Dent A, Faust K, Richer M, Djuric U, Van Ommeren R, et al
. Physician perspectives on integration of artificial intelligence into diagnostic pathology. NPJ Digit Med 2019;2:28.
Abels E, Pantanowitz L, Aeffner F, Zarella MD, van der Laak J, Bui MM, et al
. Computational pathology definitions, best practices, and recommendations for regulatory guidance: A white paper from the digital pathology association. J Pathol 2019;249:286-94.
Serag A, Ion-Margineanu A, Qureshi H, McMillan R, Saint Martin MJ, Diamond J, et al
. Translational AI and deep learning in diagnostic pathology. Front Med (Lausanne) 2019;6:185.
Aeffner F, Zarella MD, Buchbinder N, Bui MM, Goodman MR, Hartman DJ, et al
. Introduction to digital image analysis in whole-slide imaging: A white paper from the digital pathology association. J Pathol Inform 2019;10:9.
] [Full text]
Tizhoosh HR, Pantanowitz L. Artificial intelligence and digital pathology: Challenges and opportunities. J Pathol Inform 2018;9:38.
] [Full text]
Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A. Artificial intelligence in digital pathology-New tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019;16:703-15.
Acs B, Rantalainen M, Hartman J. Artificial intelligence as the next step towards precision pathology. J Intern Med 2020;288:62-81.
Ibrahim A, Gamble P, Jaroensri R, Abdelsamea MM, Mermel CH, Chen PC, et al
. Artificial intelligence in digital breast pathology: Techniques and applications. Breast 2020;49:267-73.
Koelzer VH, Sirinukunwattana K, Rittscher J, Mertz KD. Precision immunoprofiling by image analysis and artificial intelligence. Virchows Arch 2019;474:511-22.
He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019;25:30-6.
Recht MP, Dewey M, Dreyer K, Langlotz C, Niessen W, Prainsack B, et al
. Integrating artificial intelligence into the clinical practice of radiology: Challenges and recommendations. Eur Radiol 2020;30:3576-84.
Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJ. Artificial intelligence in radiology. Nat Rev Cancer 2018;18:500-10.
Dreyer KJ, Geis JR. When machines think: Radiology's next frontier. Radiology 2017;285:713-8.
Abràmoff MD, Tobey D, Char DS. Lessons learnt about autonomous AI: Finding a safe, efficacious and ethical path through the development process. Am J Ophthalmol 2020;214:134-42.
Mandal S, Greenblatt AB, An J. Imaging intelligence: AI is transforming medical imaging across the imaging spectrum. IEEE Pulse 2018;9:16-24.
Colling R, Pitman H, Oien K, Rajpoot N, Macklin P, CM-Path AI in Histopathology Working Group, et al
. Artificial intelligence in digital pathology: A roadmap to routine use in clinical practice. J Pathol 2019;249:143-50.
Ramasubbu N, Kemerer CF. Managing technical debt in enterprise software packages. IEEE Trans Softw Eng 2014;40:758-72.
Martin RC. Clean Code: A Handbook of Agile Software Craftsmanship. Upper Saddle River, New Jersey, USA: Prentice Hall; 2008.
Myers GJ. The Art of Software Testing. Hoboken, New Jersey, USA: Wiley; 2004.
Williams L, Maximilien EM, Vouk M. Test-Driven Development as a Defect-Reduction Practice. In: Proceedings of 14th
IEEE International Symposium on Software Reliability Engineering ISSRE 2003; 2003. p. 34-45.
Kothari S, Phan JH, Stokes TH, Wang MD. Pathology imaging informatics for quantitative analysis of whole-slide images. J Am Med Inform Assoc 2013;20:1099-108.
Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Atanasova K. Artificial intelligence, bias and clinical safety. BMJ Qual Saf 2019;28:231-7.
Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019;17:195.
Tellez D, Litjens G, Bándi P, Bulten W, Bokhorst JM, Ciompi F, et al
. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Med Image Anal 2019;58:101544.
Lafarge MW, Pluim JPW, Eppenhof KAJ, Veta M. Learning domain-invariant representations of histological images. Front Med (Lausanne) 2019;6:162.
Tschuchnig ME, Oostingh GJ, Gadermayr M. Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential; 2020. Available from: https://arxiv.org/pdf/2004.14936.pdf
. [Last accessed on 2020 Aug 19].
Levine AB, Peng J, Farnell D, Nursey M, Wang Y, Naso JR, et al
. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J Pathol 2020;252:178-88.
Eljasik-Swoboda T, Rathgeber C, Hasenauer R. Assessing Technology Readiness for Artificial Intelligence and Machine Learning Based Innovations. In: Proceedings of the 8th
International Conference on Data Science, Technology and Applications (Data 2019); 2019. p. 281-8.
Armstrong JR. 6.4.2 Applying Technical Readiness Levels to Software: New thoughts and Examples. In: INCOSE International Symposium; 2010. p. 838-45.
Huang CC, Lu Z. Community challenges in biomedical text mining over 10 years: Success, failure and the future. Brief Bioinform 2016;17:132-44.
Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten simple rules for a community computational challenge. PLoS Comput Biol 2015;11:e1004150.
Park SL, Pantanowitz L, Sharma G, Parwani AV. Anatomic pathology laboratory information systems: A review. Adv Anat Pathol 2012;19:81-96.
Stathonikos N, Nguyen TQ, Spoto CP, Verdaasdonk MA, van Diest PJ. Being fully digital: Perspective of a Dutch academic pathology laboratory. Histopathology 2019;75:621-35.
Pantanowitz L, Sharma A, Carter AB, Kurc T, Sussman A, Saltz J. Twenty years of digital pathology: An overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J Pathol Inform 2018;9:40.
] [Full text]
Sirota-Cohen C, Rosipko B, Forsberg D, Sunshine JL. Implementation and benefits of a vendor-neutral archive and enterprise-imaging management system in an integrated delivery network. J Digit Imaging 2019;32:211-20.
Marée R, Rollus L, Stévens B, Hoyoux R, Louppe G, Vandaele R, et al
. Collaborative analysis of multi-gigapixel imaging data using Cytomine. Bioinformatics 2016;32:1395-401.
Bankhead P, Loughrey MB, Fernández JA, Dombrowski Y, McArt DG, Dunne PD, et al
. QuPath: Open source software for digital pathology image analysis. Sci Rep 2017;7:16878.
Benson T, Grieve G. Principles of Health Interoperability: SNOMED CT, HL7 and FHIR. London, UK: Springer; 2016.
Bender D, Sartipi K. HL7 FHIR: An Agile and RESTful Approach to Healthcare Information Exchange. In: Proceedings of the 26th
IEEE International Symposium on Computer-Based Medical Systems, Porto, 2013. p. 326-331.
Herrmann MD, Clunie DA, Fedorov A, Doyle SW, Pieper S, Klepeis V, et al
. Implementing the DICOM standard for digital pathology. J Pathol Inform 2018;9:37.
] [Full text]
Osterwalder A, Pigneur Y. Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers. Hoboken, New Jersey, USA: John Wiley & Sons; 2010.
Chammassian RG, Sabatier V. The role of costs in business model design for early-stage technology startups. Technol Forecast Soc Change 2020;157:120090.
Weinberg G, Mares J. Traction: How any Startup Can Achieve Explosive Customer Growth. New York, NY: Portfolio/Penguin; 2015.
Jansen S, Bloemendal E. Defining App Stores: The Role of Curated Marketplaces in Software Ecosystems. In: International Conference of Software Business ICSOB 2013: Software Business. From Physical Products to Software Services and Solutions; 2013. p. 195-206.
APEER – Cloud-Based Digital Image Processing Platform; 2020.Available from: https://www.apeer.com
. [Last accessed on 2020 Aug 19].
Liu CZ, Au YA, Choi HS. Effects of freemium strategy in the mobile app market: An empirical study of Google Play. J Manag Inf Sys 2014;31:326-54.
Schindler RM. Pricing Strategies: A Marketing Approach. Thousand Oaks, California, USA: SAGE Publications; 2011.
Ries E. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. New York, USA: Crown Business; 2011.
Schreier J, Feeney R, Keeling P. Diagnostics reform and harmonization of clinical laboratory testing. J Mol Diagn 2019;21:737-45.
Tzankov A, Tornillo L. Hands - On experience: Accreditation of pathology laboratories according to ISO 15189. Pathobiology 2017;84:121-9.
European Commission. Guidance Document Medical Devices-Scope, Field of Application, Definition-Qualification and Classification of Stand Alone Software-MEDDEV 2.1/6; 2016. Available from: https://ec.europa.eu/docsroom/documents/17921
. [Last accessed on 2020 Jul 01].
Lamph S. Regulation of medical devices outside the European Union. J R Soc Med 2012;105 Suppl 1:S12-21.
Gupta SK. Medical device regulations: A current perspective. J Young Pharma 2016;8(1):6–11.
European Commission. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on Medical Devices, Amending Directive 2001/83/EC, Regulation (EC) no 178/2002 and Regulation (EC) no 1223/2009 and Repealing Council Directives 90/385/EEC and 93/42/EEC; 2017. Available from: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017R0745&from=EN
. [Last accessed on 2020 Apr 06].
García-Rojo M, De Mena D, Muriel-Cueto P, Atienza-Cuevas L, Domínguez-Gómez M, Bueno G. New European Union regulations related to whole slide image scanners and image analysis software. J Pathol Inform 2019;10:2.
International Organization for Standardization. ISO 13485:2016 Medical Devices - Quality Management Systems - Requirements for Regulatory Purposes; 2016. Available from: https://www.iso.org/standard/59752.html
. [Last accessed on 2020 Jul 01].
International Organization for Standardization. ISO 14971:2019 Medical Devices-Application of Risk Management to Medical Devices; 2019. Available from: https://www.iso.org/standard/72704.html
. [Last accessed on 2020 Jul 01].
International Electrotechnical Commission. IEC 62304:2006 Medical Device Software-Software Life Cycle Processes; 2006. Available from: https://www.iso.org/standard/38421.html
. [Last accessed on Jul 01].
Proscia. DermAI: Streamline High-Volume Derm Workflows; 2020. Available from: https://proscia.com/ai/
. [Last accessed on 2020 Apr 30].
LeRoy L, Kalfoglou AL, Wolman DM, editors. Medicare Laboratory Payment Policy: Now and in the Future. Washington, DC, USA: National Academies Press; 2001.
Kassenärztliche Bundesvereiningung. Einheitlicher Bewertungsmaßstab (1. Juli 2020); 2020. Available from: https://www.kbv.de/html/ebm.php
. [Last accessed on 2020 Aug 19].
Samson D, Schoelles KM. Chapter 2: Medical tests guidance (2) developing the topic and structuring systematic reviews of medical tests: Utility of PICOTS, analytic frameworks, decision trees, and other frameworks. J Gen Intern Med 2012;27 Suppl 1:S11-9.
Food and Drug Administration, Centers for Medicare and Medicaid Services. Program for parallel review of medical devices. Fed Regist 2016;81:73113-5.
Klug SJ, Neis KJ, Harlfinger W, Malter A, König J, Spieth S, et al
. A randomized trial comparing conventional cytology to liquid-based cytology and computer assistance. Int J Cancer 2013;132:2849-57.
Alvi E, Gupta R, Borok RZ, Escobar-Hoyos L, Shroyer KR. Overview of established and emerging immunohistochemical biomarkers and their role in correlative studies in MRI. J Magn Reson Imaging 2020;51:341-54.
Hamilton PW, Bankhead P, Wang Y, Hutchinson R, Kieran D, McArt DG, et al
. Digital pathology and image analysis in tissue biomarker research. Methods 2014;70:59-73.
Molin J. Diagnostic Review with Digital Pathology: Design of Digitals Tools for Routine Diagnostic Use. PhD Thesis. Department of Computer Science; Engineering, Chalmers University of Technology; 2016. Available from: https://books.google.de/books?id=MTA4vwEACAAJ
. [Last accessed on 2020 Aug 19].
Small Business Innovation Research (SBIR) Program; 2020. Available from: https://www.sbir.gov
. [Last accessed on 2020 Aug 19].
Guo H, Birsa J, Farahani N, Hartman DJ, Piccoli A, O'Leary M, et al
. Digital pathology and anatomic pathology laboratory information system integration to support digital pathology sign-out. J Pathol Inform 2016;7:23.
] [Full text]
Ecosystem for Pathology Diagnostics with AI Assistance; 2020. Available from: https://www.empaia.org
. [Last accessed on 2020 Aug 19].
Marble HD, Huang R, Dudgeon SN, Lowe A, Herrmann MD, Blakely S, et al
. A regulatory science initiative to harmonize and standardize digital pathology and machine learning processes to speed up clinical innovation to patients. J Pathol Inform 2020;11:22. [Full text]
Wiegand T, Krishnamurthy R, Kuglitsch M, Lee N, Pujari S, Salathé M, et al
. WHO and ITU establish benchmarking process for artificial intelligence in health. Lancet 2019;394:9-11.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5]