Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 196  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 




 
Table of Contents    
RESEARCH ARTICLE
J Pathol Inform 2014,  5:33

Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization


1 Department of Dermatology, School of Medicine, University of Pittsburgh, Pittsburgh, USA
2 International and Commercial Services Division, UPMC, Pittsburgh, USA
3 Omnyx LLC, Pittsburgh, USA
4 Office of Sponsored Programs and Research Support, University of Pittsburgh Medical Center, UPMC, Pittsburgh, PA, USA
5 Department of Pathology, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA

Date of Submission28-Mar-2014
Date of Acceptance22-Jun-2014
Date of Web Publication28-Aug-2014

Correspondence Address:
Jonhan Ho
Department of Dermatology, School of Medicine, University of Pittsburgh, Pittsburgh
USA
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/2153-3539.139714

Rights and Permissions
   Abstract 

Background: Digital pathology offers potential improvements in workflow and interpretive accuracy. Although currently digital pathology is commonly used for research and education, its clinical use has been limited to niche applications such as frozen sections and remote second opinion consultations. This is mainly due to regulatory hurdles, but also to a dearth of data supporting a positive economic cost-benefit. Large scale adoption of digital pathology and the integration of digital slides into the routine anatomic/surgical pathology "slide less" clinical workflow will occur only if digital pathology will offer a quantifiable benefit, which could come in the form of more efficient and/or higher quality care. Aim: As a large academic-based health care organization expecting to adopt digital pathology for primary diagnosis upon its regulatory approval, our institution estimated potential operational cost savings offered by the implementation of an enterprise-wide digital pathology system (DPS). Methods: Projected cost savings were calculated for the first 5 years following implementation of a DPS based on operational data collected from the pathology department. Projected savings were based on two factors: (1) Productivity and lab consolidation savings; and (2) avoided treatment costs due to improvements in the accuracy of cancer diagnoses among nonsubspecialty pathologists. Detailed analyses of incremental treatment costs due to interpretive errors, resulting in either a false positive or false negative diagnosis, was performed for melanoma and breast cancer and extrapolated to 10 other common cancers. Results: When phased in over 5-years, total cost savings based on anticipated improvements in pathology productivity and histology lab consolidation were estimated at $12.4 million for an institution with 219,000 annual accessions. The main contributing factors to these savings were gains in pathologist clinical full-time equivalent capacity impacted by improved pathologist productivity and workload distribution. Expanding the current localized specialty sign-out model to an enterprise-wide shared general/subspecialist sign-out model could potentially reduce costs of incorrect treatment by $5.4 million. These calculations were based on annual over and under treatment costs for breast cancer and melanoma estimated to be approximately $26,000 and $11,000/case, respectively, and extrapolated to $21,500/case for other cancer types. Conclusions: The projected 5-year total cost savings for our large academic-based health care organization upon fully implementing a DPS was approximately $18 million. If the costs of digital pathology acquisition and implementation do not exceed this value, the return on investment becomes attractive to hospital administrators. Furthermore, improved patient outcome enabled by this technology strengthens the argument supporting adoption of an enterprise-wide DPS.

Keywords: Anatomic pathology, cost, cost analysis, digital pathology, productivity, whole slide imaging


How to cite this article:
Ho J, Ahlers SM, Stratman C, Aridor O, Pantanowitz L, Fine JL, Kuzmishin JA, Montalto MC, Parwani AV. Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization. J Pathol Inform 2014;5:33

How to cite this URL:
Ho J, Ahlers SM, Stratman C, Aridor O, Pantanowitz L, Fine JL, Kuzmishin JA, Montalto MC, Parwani AV. Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization. J Pathol Inform [serial online] 2014 [cited 2017 Dec 12];5:33. Available from: http://www.jpathinformatics.org/text.asp?2014/5/1/33/139714


   Introduction Top


Digital pathology includes the integration of whole slide images (WSIs) into pathologists' clinical workflow by offering them a digital manner to manage, interpret, analyze, and archive pathology information. One of the main purposes of digital pathology is to enable pathology workflow, while maintaining or improving interpretive accuracy.

Workflow improvement benefits as a result of employing digital pathology include the support of lab automation (e.g. bar coding and tracking of assets, bidirectional interfaces with the scanners) and the potential to increase an individual pathologist's productivity by at least 13% due to improved organization and tracking of surgical pathology cases. [1] The 13% estimate was obtained by performing a time and motion study, whereby researchers observed pathologists performing their daily tasks and recorded the amount of time it took to complete the tasks. Six pathologists were observed over 12 days of work. The average time on case work per day was ~5 h and 20 min. The authors identified several categories of opportunities for time savings; matching slides and paperwork to cases, error correction, such as obtaining correct or missing paperwork, retrieving prior cases, transporting cases, organizing cases, querying for cases such as checking for new cases or stat cases, searching for specific cases such as when fielding a call about a specific case from a clinician, and communication, such as sending region of interest images instead of co-scheduling time to examine the case at a multi-headed scope. About 36.0% of the pathologists' recorded time was spent actually reviewing slides, 34.6% of time spent reporting, 13.4% performing tasks that may be easier or automated with digital pathology, and 16% of time spent performing other tasks (such as research related tasks). Of note, the environment pathologists were observed in utilized a modern lab, already equipped with an electronic specimen and slide tracking system including barcoding. Outside of the 13.4%, the study mentioned additional unobserved opportunities for time saving, including preparation for tumor boards and consultations. In addition, nonattending pathologists, including trainees as well as ancillary staff, were also observed performing tasks such as matching and preparing cases for review, which were not included in the 13.4%. Digital distribution of cases also offers easy sharing of slides throughout the health care organization, thereby allowing distribution of work among a network of pathologists, regardless of their geographic location. This allows for real-time workload leveling (i.e. workload distribution) and ensures that pathologists across the organization are fully and evenly utilized. It also enables and facilitates the occurrence of remote secondary consultations by pathology subspecialists. As digital pathology decouples pathologists from the histology laboratory and conventional microscopes, implementation of this technology will also facilitate centralization of pathology services, thereby reducing the number of histopathology labs needed and promote more efficient lab staffing models.

Interpretive accuracy improvement benefits include getting the right expert pathologist to view difficult cases, use of computer assisted quantitative tools to perform image analysis, and support of quality assurance (QA) and education programs. [2] Pathologist interobserver interpretive variability is a factor often cited for contributing to diagnostic errors. [3] Differences in experience between general pathologists and subspecialty pathologists may account for a portion of this diagnostic error rate. [4],[5],[6] Implementation of a digital pathology system (DPS) at remote sites where community pathologists are located may allow them to have easier access to subspecialty pathologists. This, in turn, can lead to a reduction in interpretive errors and improve patient outcomes.

While digital pathology is already commonly used for research and education, its clinical use has been limited to niche low volume applications such as interpreting remote frozen sections and secondary consultations. [7],[8],[9] Apart from Food and Drug Administration (FDA) regulatory constraints to utilize WSI for primary diagnosis in pathology in the USA, the dearth of financial accounting data supporting a return on investment prior to purchasing expensive WSI scanners and supporting software is a major factor delaying wide-scale adoption of digital pathology. Digital pathology implementation carries high acquisition costs and involves other extra costs. The latter includes additional histopathology and IT personnel to conduct scanning and support the IT-based DPS, as well as extra IT costs to support integration with other IT-based medical devices and systems. Digital pathology is considered disruptive to current histology workflow due to its inability to eliminate the glass slide preparation step and therefore introduction of an additional step and cost to the surgical pathology workflow. [10] Digital pathology will be more likely to be adopted on a large scale if it will offer a clear economic benefit. Only a limited number of studies were conducted to determine the economic impact of WSI system implementation. A recent study evaluated the economic impact of WSI implementation at a large pathology practice within an academic based institution, focusing on a "value added" approach. This approach evaluated potential cost savings, time savings and improved quality of service across various applications that may benefit from digital pathology - patient care (i.e. clinical use), as well as education and research. [11] Impact of WSI implementation was demonstrated by outcome-based measures such as number of scanned slides, pathologist acceptance, and expanded utilization for new clinical applications.

The institution where the analysis for this study was conducted is a large academic medical center-based health care organization expecting to fully adopt digital pathology upon its regulatory approval for primary diagnostic use. An economic model was developed to help estimate and quantify the potential financial savings due to enterprise-wide implementation of a DPS throughout the health care organization, focusing on patient care-related activities (i.e. clinical use) and benefits. The economic model was based on a cost savings that could be achieved through implementation of a DPS throughout the organization focused on improved workflow and diagnostic capability.


   Methods Top


This analysis was developed as an economic impact model for a pathology department that employs a network of pathologists located at both academic and community-based hospitals and distributed across a large geographical area. Within the academic hospitals, the pathology department utilizes a "centers of excellence" model (i.e. case sign-out by subspecialists only, limited to one surgical pathology subspecialty), while at the smaller community hospitals case sign-out is conducted primarily by general pathologists and applied across multiple organ systems.

Context/Setting

Health Care Pathology Department Infrastructure


The institution is a large academic-based medical center and health care organization in Western Pennsylvania, operating more than 20 academic, community and specialty hospitals and 400 outpatient sites. The institution serves more than 264,000 in-patient admissions and observation cases, more than 3.6 million outpatient visits and around 174,000 surgeries annually. The institution also carries the largest health insurance plan within Western Pennsylvania, thereby serving as an integrated health care organization, playing a dual role as both health care provider and payor.

Each hospital in the health care organization that is equipped with a histology lab currently has at least one pathologist on site to facilitate standard turn-around time for more routine sign-out of surgical cases/specimens, and also to ensure support for real-time interpretation of frozen sections during surgeries conducted at that hospital. Each community hospital has a histopathology laboratory located within the hospital that processes and prepares slides and cases for the local hospital pathologists' review and interpretation.

The pathology department within the organization utilizes the "centers of excellence" model within its academic hospitals. At the community hospitals, general, nonsubspecialist pathologists (generalists) review and provide diagnoses for multiple organ systems. Occasionally, generalists may request a formal second opinion from subspecialists located at the academic centers.

Economic Impact Model

The cost savings analysis was conducted for a 5-year roll out implementation intended to support the institution's planned transition from the current analog-based surgical pathology system composed of optical microscopes and glass slides to an enterprise-wide use of a DPS that will employ digital slides. Cost savings estimates were based on two main benefits associated with the use of digital pathology: (1) Potential improvements in workflow/productivity and lab consolidation; and (2) avoided treatment costs due to reduced rates of interpretive errors by general, nonsubspecialist pathologists (generalists) within the institution.

Data Collection

Histopathology laboratory data (e.g. number of accessions per each lab, lists and numbers of accessions for various cancer types) and personnel data (e.g. number of pathologists and histotechnologists at each lab and hospital) were collected for calendar year 2012. Data were collected via interviews with pathologists, physicians, and operations managers within the health care organization.

Assumptions

The analysis was based on the following assumptions.

Clinical Use

A transition from no use of digital pathology to a complete digital pathology practice will occur and will be utilized for routine surgical pathology primary diagnosis throughput the entire health care organization (pending FDA approval for use of WSI for surgical pathology primary diagnosis).

Equipment and IT Infrastructure

(1) Whole slide image scanners and supporting software will be purchased and placed in each of the current histopathology laboratories throughout the entire health care organization. The number of scanners will be based on current specimen volume at each lab and scan time per slide. Scan time was projected to decrease throughout the 5-year plan from 140 seconds/slide to 69 seconds/slide due to projected improvements in WSI scanner technology. Accompanying WSI workstations (i.e. computer monitors supporting image management and image viewing software) will be purchased and placed within pathology offices throughout the entire health care organization. The majority of WSI systems will be purchased in year 1 of implementation, with additional scanners and workstations added to account for future volume growth and increased utilization in subsequent years [Table 1]; (2) the DPS will be fully integrated with the Anatomic Pathology Laboratory Information System (APLIS) and electronic medical records.
Table 1: Acquisition of WSI scanners and workstations and anticipated adoption rates: 5-year roll out plan

Click here to view


Digital Pathology Adoption/Utilization of Whole Slide Image Systems

The digital pathology adaption curve, that is the conversion from the existing analog paradigm of microscopes and histopathology glass slides to DPSs/digital slides, will start at approximately 25% in year 1 of implementation. The adoption rate will then increase in subsequent years to reach 90% by year 4 [Table 1]. The dual modality of optical microscopes/glass slides and digital pathology/digital slides will co-exist for several years.

Lab Consolidation

Consolidation of current hospital-based histopathology laboratories into two main pathology laboratories within the organization will occur at years 2-4 of DPS implementation.

Reduction in Cancer Interpretative Error Rates by Nonsubspecialized Pathologists

Use of an enterprise-wide DPS will decrease the rate of interpretation errors conducted by nonsubspecialized pathologists by changing the distributed sign-out model: The DPS will support and enhance the ability of all nonsubspecialized pathologists to develop proficiency in one or several subspecialties. This can occur as the DPS will allow for aggregate volumes of specimen/cases for a single organ system from across the health care organization to be available for viewing and interpretation by any pathologist within the health care organization, regardless of location. In comparison, in the current localized sign-out model, each hospital may not accumulate enough volume of a single specific organ type (e.g. breast pathology, dermatopathology) to justify employing a subspecialist on site. Reduction in cancer interpretive error rates are assumed to start at 15% from year 1, and increase to 30, 50, 75%, and 75% at subsequent years.

Payor/Provider Cost Sharing and Savings

As an integrated health care provider/payor organization, an increasing trend within the health care market, [12] the integrated organization is even further incentivized to deliver health care efficiently over a large population. Anticipated changes in the health care market in the near future are estimated to increase the burden of patient care for both providers and payors, thereby increasing financial incentives to reduce patient care costs. Therefore, for an integrated health care organization, savings offered by an enterprise-wide DPS implementation can be shared by the provider and payor parts of the organization. Sharing of these savings to the integrated organization is estimated to increase from 21% in year 1 to 25% in year 2, and reach a maximal rate of 50% at year 3.

Operational Cost Savings

Gains in Workflow/Productivity and Lab Consolidation and their Related Cost Savings


Increased efficiencies were reflected as gains in workflow/productivity for pathologists and histotechnologists, calculated as clinical full-time equivalent (FTE) capacity gains. One FTE was defined as one employee (pathologist or histotechnologist) working full-time during a fixed time period (i.e. 1 year of working hours).

Pathologist productivity gains were based on three main components: Productivity improvements for individual pathologists within the organization, productivity improvements for the pathology department as a whole, and a reduction in the number of second opinion consults provided by subspecialists to general pathologists within the organization (i.e. internal consults). Productivity improvements for an individual pathologist were based on an improved organization and tracking of surgical pathology cases, as shown in a recent time and motion study to be at 13%. [1] As academic pathologists dedicate significant time to research activities, pathologist FTE gains were adjusted to reflect only time dedicated to the clinical activities of each pathologist (i.e. clinical FTE gains), followed by an adjustment for the adoption rate of digital pathology throughout the 5-year roll out plan. Productivity improvements for the pathology department as a whole, that is level loading (or balancing workload), were based on the estimated increase in utilization of pathologist capacity due to the potential of an enterprise-wide DPS to enable workload distribution across the organization. Pathologist FTE gains due to level loading were calculated for labs with a current workload that is lower than the average workload within the organization. Implementation of an enterprise-wide DPS was estimated to reduce the number of second opinion consult requests from generalists to subspecialists within the health care organization due to an expected increase in experience of the generalists. FTE gains were calculated based on a 50% reduction rate of the current number of internal consults, adjusted according to the digital pathology adoption rate.

Histotechnologist productivity gains were anticipated due to the potential consolidation of laboratories within the organization, starting at year 2 of implementation. Gains in histotechnologist FTE were expected due to the reduced number of histotechnologists to be employed throughout the organization following laboratory consolidation. Economies of scale gained through laboratory consolidation (migration of technical work from many labs in each hospital into centralized laboratories within a region) would enable staffing to match the sum total of needs throughout the entire organization rather than hiring for the needs of separate histology labs individually. For example, if one site needed 2.5 FTE's, and hired 3 FTE's, and another site needed 4.5 FTE's and hired 5 FTE's, lab consolidation would enable hiring 7 FTE's to match needs instead of overstaffing at 8 FTE's when lab needs were taken individually. Digitization also allows labs to consolidate without the turnaround time delays of shipping twice as well as automates the logistics of sorting, packaging, and transport that can be a bottleneck.

Pathologist clinical FTE capacity gains were translated into potential cost savings offered due to avoided future hiring of pathologists within the organization required to "catch up" with the anticipated increase in the number of accessions and anticipated attrition. The cost basis for pathologist FTE capacity gains was based on the department of pathology historical attrition rate and the annual cost of employing pathologists (including salary and benefits) [Table 2]. The cost basis for histotechnologist FTE capacity gains was based on the annual cost of employing a histotechnologist [Table 2].
Table 2: Productivity, lab consolidation and improved interpretive accuracy: Assumptions, values and cost basis

Click here to view


An additional component, avoided cost of additional new optical microscopes which would have been purchased to replace aging microscopes, was included to determine the total cost savings offered due to increased productivity gains and laboratory consolidation savings.

Improved Outcome: Avoided Treatment Costs

Analysis was conducted to estimate current extra costs within the organization acquired due to pathologist diagnostic errors (not including other pathology/histology related errors such as specimen mislabeling, etc.). These incremental costs were estimated to serve as avoided treatment costs offered by an improved and more accurate diagnosis enabled by the enterprise-wide implementation of a DPS. Analysis focused on the interpretive errors conducted by nonsubspecialized pathologists within the pathology department for 12 common cancers. Estimates did not include interpretive errors due to noncancer diagnoses.

Detailed estimates of avoided treatment costs due to interpretive errors within the organization were initially conducted for breast and melanoma cancers. Calculations were conducted for over treatment and under treatment costs due to false positive (overcall) and false negative (under call) interpretive errors of these two cancers, respectively. Breast cancer was chosen because this cancer has a high public awareness. Melanoma was used as the other example because this cancer has a notoriously high rate of false negative diagnoses. [13]

These detailed estimated total annual costs for breast cancer and melanoma interpretive errors by nonsubspecialists pathologists within the organization were then averaged to establish the average total annual extra costs of cancer errors per each cancer case. The average cost was extrapolated to determine the total annual costs of interpretive errors by nonsubspecialists pathologists for an additional 10 common cancers type (defined as the 10 most common cancers in the USA by the Center for Disease Control [CDC]). [14]

For each of the 12 cancer types, the total number of surgical pathology cases for a single tissue type obtained at community hospitals during calendar year 2012 (indicated as the number of biopsy parts listed within the APLIS, but limited to parts that were submitted to rule out a cancer diagnosis), was collected. The estimated rate of cancer diagnosis (i.e. sign-out as cancer diagnosis) among the total biopsy parts that were submitted to rule out a cancer diagnosis per each tissue type were provided by the institution's subspecialty pathologists. For melanoma calculations, the estimated rate of a potential cancer diagnosis was used, as not all skin specimens were submitted to rule out skin cancer (or specifically melanoma). Interpretive error rates per each cancer type (categorized by tissue type) were based on rates published by Raab and Grzybicki [3] except for breast cancer (maximal error rate estimate provided by the institution's breast subspecialty pathologist based on Price et al. [15] ). Raab and Grzybicki study rates were based on major discrepancy rates reported from multiple interinstitutional pathology slide review studies. [3] For melanoma, the rate provided by Raab and Grzybicki was adjusted to reflect the effective interpretative error rate.

Melanoma Errors

Detailed treatment cost calculations were estimated for both over and under treatment of melanoma due to false negative and false positive interpretive errors, respectively. A false negative error and its potential incremental treatment cost due to progression of early melanoma (stage 0, melanoma in situ) to Stage I, II, and III melanoma due to under treatment were calculated based on a study by Alexandrescu. That study used 2008 Mid Atlantic Medicare costs and calculated total melanoma costs per clinical tumor (T) stage. [16] Estimated rates of progression and recurrence of early melanoma (stage 0, melanoma in situ) at more advanced American Joint Committee of Cancer (AJCC) stages were provided by dermatopathologists at our health care organization. Calculations for false positive interpretive errors, leading to unnecessary treatment costs (i.e. over treatment) were also based on costs for early disease (i.e. melanoma in situ) as published by Alexandrescu. [16]

Breast Cancer Errors

A similar analysis for over treatment costs due to false positive interpretive errors of breast cancer was conducted. Detection of early breast disease was defined as a diagnosis of atypical ductal hyperplasia (ADH). The total number of breast biopsies collected at community hospitals during calendar year 2012 and submitted to rule out breast cancer was collected. Using the interpretative error rate for breast cancer (a maximal error rate estimate based on Price et al. [15] ), the total annual number of cases with interpretive errors within community hospitals was determined. Based on 2013 Medicare reimbursement costs, [17] the total annual cost of unnecessary treatment for breast cancer errors was determined.

Another 10 Common Cancer Errors

The detailed annual total costs calculated for breast cancers and melanoma errors within community hospitals were combined, averaged and then extrapolated for additional 10 common cancer types in the USA. The list of common cancer types was based on data provided by the CDC). [14]

The estimated annual incremental treatment costs caused by interpretive errors for each of the 12 most common cancer types was calculated as the product of: The total annual number of accessions per tissue type submitted to rule out cancer at community histopathology labs (a); estimated cancer diagnosis rates among total accessions for tissue type (estimates provided by our institution subspecialty pathologists) (b); rates of the interpretive error for that cancer (based on published studies) [3] (c); and annual costs of an interpretive error per cancer case (d) (i.e. a × b × c × d). For melanoma and breast cancer, d value was based on the detailed analysis mentioned above. For the other 10 common cancers, d value was based on the weighted average annual cost for interpretive error case (extrapolated from calculations used for melanoma and breast cancer).

A sum of estimated incremental annual treatment costs for each of the 12 most common cancer types was used as the basis for estimating the potential total annual avoided treatment costs. To reach the actual annual savings for the institution throughout the 5-year implementation plan, the potential total annual avoided treatment savings (a) were adjusted according to the adoption rate of digital pathology (b), the anticipated reduction in errors by nonsubspecialized pathologists (c), and the anticipated health care organization share of savings (d) at each year (i.e. a × b × c × d).


   Results Top


Health Care Organization

Pathology Department Infrastructure


[Table 3] provides details for case and slide volumes and pathologist/histotechnologist FTE's per hospital in calendar year 2012. In aggregate, the health care organization received and processed about 219,000 accessions (i.e. cases) and created about 1.7 million glass slides. Approximately 54% (118,000/219,000) of surgical cases were collected, processed, and interpreted at academic hospitals. The annual accession growth rate has been about 2.5%.
Table 3: Histopathology infrastructure (calendar year 2012 site profile)

Click here to view


Of a total of 94 pathologists, 66% were located at academic centers, while 33% were located at community hospitals. Following adjustment for time dedicated to research activities, the total combined clinical pathologist FTE's in the organization (based solely on time dedicated to clinical activity) was 66. The annual average ratio of surgical cases per pathologist (dedicated to clinical activity) was about 3300. The organization was staffed by 82 histotechnologist FTE's (academic labs, n = 50; community labs, n = 32).

Savings Offered by Gains in Productivity and Lab Consolidation

[Table 4] presents the estimated gains in productivity for individual pathologists and the pathology department as a whole and the relevant cost savings offered by DPS implementation. Productivity improvements, estimated at the rate of 13%, [1] translated into an approximate increase of 1.5 cases per workday in FTE for pathologists. In actual practice, this number will be higher since pathologists do not sign-out cases every day and will greatly vary between "benches" (i.e. surgical pathology services structured according to pathology organ systems) that have a high volume of cases and fewer slides per case (i.e. biopsies) and those that have a low volume of cases, but more slides per case (i.e. large tumor resections). The total savings due to increased pathologist productivity during the planned DPS 5-year roll out reflected savings offered due to avoiding future hiring of pathologists based on an anticipated future growth in the number of accessions. These savings represented the major component of savings, accumulating to a total of over $10 million.
Table 4: Productivity savings: 5-year roll-out (in $ thousands)

Click here to view


Implementation of a DPS would also enable, over time, centralization of the histopathology lab operations with more efficient staffing models. The 5-year total avoided costs due to laboratory consolidation and reduction of histotechnologist labor was estimated at $1.46 million [Table 4]. Including the avoided purchase of microscopes, the total 5-year productivity savings offered by DPS implementation were estimated at $12.38 million.

Savings Offered by Improved Interpretive Accuracy

[Table 5] demonstrates the detailed calculation steps conduced to estimate the total number of pathologist interpretive errors and their related extra treatment costs for melanoma and breast cancer.
Table 5: Cost quantification of melanoma and breast cancer interpretive errors

Click here to view


Melanoma Errors: Cost of Under Treatment and Over Treatment

We analyzed the potential savings in melanoma diagnosis and treatment at our institution using a published interinstitutional major interpretative discrepancy rate of 2.3% for melanoma. [3] Based on polled dermatopathologists, false negative interpretive errors of melanoma or atypical lesions that should be treated as melanoma typically accounted for 90% of total interpretive errors [13] (therefore representing 2.1% of total melanoma interpretive errors cases). False positive diagnoses typically accounted for 10% (therefore representing 0.23% of total melanoma interpretive error cases). Based on polled dermatopathologists at our institution, approximately 90% of all of those lesions had "clean margins" (i.e. the lesion was completely excised); thereby, a false negative error typically did not have a detrimental impact on outcome. About 10% of false negative errors may represent patients with residual melanoma and therefore at risk for disease recurrence and may lead to additional, more costly treatment. Therefore, the effective false negative error rate (i.e. error that may affect treatment) was determined at 0.21%. Combined with a false positive error rate at 0.23%, the overall effective melanoma interpretive error rate was 0.44%.

An analysis of all skin specimens reviewed by general, nonsubspecialized pathologists during the 2012 calendar year was conducted. Of a total of 7,662 skin specimens collected and interpreted by general pathologists, our institution's dermatopathologists estimated that 40% of specimens were performed to rule out melanoma (3,065/7,662). Based on a false negative interpretive error rate for melanoma at 0.21%, a total of six melanoma cases per year (0.0021 × 3,065) were estimated to represent false negative errors that progressed to more advanced stages, thereby requiring more costly and invasive treatment. The incremental annual cost due to this avoidable progression, calculated for risk of progression to various disease stages (melanoma AJCC Stage I, II and III) was estimated at $109,448 (approximately $17,500 per melanoma false negative error case). An analysis was also conducted for false positive errors of melanoma/atypical lesions. Assuming that 100% of these cases were thereby unnecessary treatments, using an effective error rate of 0.23% revealed that a total of seven cases per year could have represented false positive errors. As the incremental cost of treating false positive melanoma errors as early disease was approximately $5,044 per error case, the total annual cost was estimated at $35,300. In summary, the total estimated incremental and avoidable annual costs for melanoma errors by nonsubspecialist pathologists, including both false negative and false positive diagnoses (n = 13.4), was approximately $144,756. Therefore, the estimated average incremental annual cost was approximately $10,803 per melanoma error.

Breast Cancer Errors: Cost of Over Treatment

Representing over treatment costs due to a false positive diagnosis, breast cancer interpretive error was analyzed [Table 5]. Of the total annual number of breast biopsies (typically removed due to suspected ADH) that were interpreted by nonsubspecialized pathologists, a breast cancer pathology subspecialist estimated that 10% (184/1,838) were diagnosed as ADH. The subspecialist also estimated that approximately a maximum of 15% of ADH diagnoses (28/184) were false positive diagnoses. Based on 2013 Medicare reimbursement rates for ADH (defined as Stage 1A, 1+ human epidermal growth factor receptor 2 score), the estimated imputed annual cost of unnecessary treatment of breast cancer error case was $26,611.

Most Common Cancers Errors: Costs

The aforementioned calculations for melanoma and breast cancer were extrapolated to a broader cohort of cancers. Based on the above weighted average cost of over and under treatment, an annual weighted average cost of about $21,500/cancer interpretation error case was suggested for another 10 common cancer types [Table 5].

Based on the annual volume of accessions interpreted by nonsubspecialist pathologists at our institution, published major interpretive error rates for each cancer type, and the estimated cancer diagnosis rate among specimens collected for a single tissue organ within the organization, the estimated annual incremental treatment costs due to errors for the 12 most common cancers were estimated [Table 6]. The total annual cost of cancer errors by nonsubspecialist pathologists within the health care organization could be as high as $5.88 million.
Table 6: Estimated incremental annual treatment costs due to interpretive errors for 12 most common cancersa ($ in thousands)

Click here to view


[Table 7] describes the estimated potential annual avoided treatment costs due to improved diagnoses upon implementing a DPS throughout a 5-year roll out plan. For each year, the annual costs of $5.88 million were adjusted to reflect the adoption rate of digital pathology, the anticipated reduction in error rate by nonsubspecialized pathologists. Annual savings increased each subsequent year as adoption rates increased. Unrelated factors, such as the anticipated increased share of savings due to the changing health care environment, further contributed to an increase in savings. Major annual savings were estimated to start to occur at year 2 at $0.44 million, growing to annual savings of $1.98 million at year 4. Total avoided cost throughout the 5-year DPS roll out plan, based on improved diagnostic accuracy and its related reduced errors and thereby avoided cancer treatment expenses was estimated at $5.35 million.
Table 7: Avoided incremental treatment costs throughout the five-year DPS implementation roll-out for 12 most common cancersa ($ in thousands)

Click here to view


Total Operational Cost Savings

The total 5-year operational cost savings offered by DPS implementation are summarized in [Table 8]. Operational savings were based on improved productivity and improved interpretive accuracy leading to improved diagnoses and improved patient care. The 5-year cost savings were mainly due to improved productivity, estimated at $12.38 million. Cost savings due to improved diagnoses, leading to avoidance of unnecessary treatment costs were estimated at $5.35 million. Total savings throughout the 5-year DPS roll out were suggested at $17.73 million.
Table 8: Digital pathology use case: Total cost savings (in $ thousands)

Click here to view



   Discussion Top


To justify and support transition of pathology labs and health care organizations to digital pathology, a detailed cost-benefit analysis should be conducted. Calculations of the costs involved in acquisition of a DPS composed of hardware (i.e. WSI scanners) and supporting software should be weighed against the potential economic savings offered by its use. Therefore, we developed an economic impact model to help estimate potential economic benefits of implementing DPS for routine primary diagnosis (assuming FDA approval). Our economic impact model, as presented in this report, was based on and limited to two main benefits offered by implementation of an enterprise-wide DPS at a large and geographically distributed pathology department within a large health care organization: (1) Productivity/efficiency improvements; and (2) improved outcomes derived from a reduced rate of surgical pathology interpretive errors.

The transition from conventional light microscopy to a digital-based workflow for imaging potentially offers improved productivity/efficiency and related operational cost savings. For example, following the implementation of digital radiology, utilizing a Picture Archiving and Communications System, productivity improvement rates of 12-18% were reported in radiology departments adopting digital radiology. [18],[19] In anatomic or surgical pathology, the transition to a WSI system could result in a similar efficiency, with an improvement rate of 13% for individual pathologist when compared with a manual-based system utilizing conventional microscopes and glass slides. [1] Increased efficiency could be due to an improvement in tasks such as organizing cases, querying and searching for cases, and "switching costs" from having to suspend analysis, while waiting for delivery. [1] However, this data remains theoretical and unproven, and does not take into account possible inefficiencies that a WSI system might introduce, such as adding an extra step in the workflow, thereby introducing extra tasks such as glass slide scanning tasks into the workflow. One advantage of digital pathology over the current practice model is that case workload can be distributed across a greater group of pathologists, independent of their physical location, enabling more efficient utilization of pathologists throughout the pathology department. Implementation of an enterprise-wide DPS also enables a greater degree of centralization of histopathology labs throughout the organization, promoting economies of scale in laboratory operation. The cost savings offered by increasing the surgical pathology workflow efficiency were the main component of the potential savings offered by an enterprise-wide DPS. The 5-year cost savings due to improved productivity were estimated at $12.38 million, composing 70% of total savings offered by implementation of a DPS. While not all health systems are the same, it is conceivable that when these numbers are extrapolated to a smaller number of total accessions, a system similar to, but with half our volume (109,500 total accessions) might realize savings of $6.19 million, and a system with a quarter of the volume, $3.1 million.

The second main component of cost savings is supporting improved patient outcomes by reducing interpretive errors and thereby eliminating costs for unnecessary treatment or progression of a disease and its related extra costs. However, these costs may be overestimated due to the limitations of our model. First, the interpretive error rates that were used in our analysis were based on published reports that were mainly conducted at reference centers. These centers typically review a higher proportion of cancer cases compared with other pathology practices, thereby possibly inflating actual error rates. Second, our calculation of those errors was conducted exclusively for nonsubspecialists within the organization. However, the quoted studies for interpretive errors used in this analysis used different and various methodologies and generally did not distinguish whether referrals came from subspecialists or nonsubspecialists. Therefore, our model may overestimate the cost savings due to improved patient outcomes due to reduced interpretative errors by nonsubspecialists within the pathology department. In addition, our institution has implemented many QA protocols, including presign-out QA protocols, which have helped reduce errors, [20] but are not included in our assumptions. In the referenced study, a random 5% presignout QA program was put into place in the core academic hospitals, which found a disagreement of 2.3% when including minor (2.2%), moderate (0.1%), and major (0%) disagreements. It should be noted that community hospitals were not included in the referenced study. The rate of error reduction in the current study was arbitrarily estimated to be from 15%-30%-50%-75%-75% over 5 years resulting in savings of $5.35 million. A more conservative estimate of 5%-10%-20%-30%-50% would result in savings of $ 2.65 million over 5 years. It should be noted that the error reduction percentage is a subset of DPS utilization, and not a reduction of the overall number of cases.

However, there were also potential reasons for underestimation. Most notably, several subspecialties were not included in the analysis, since they were not included in the 12 most common cancers. Many specimen subsets were also not included in our analysis, such as skin biopsies for any other diseases except for pigmented lesions. Our analysis may also be relevant for a limited and small number of pathology departments supporting a similar large and distributed health care organization with a similar patient demographics profile. An additional limitation is that our final calculations were adjusted for share of cost savings for our health care organization, an integrated provider/payer organization. Therefore, our model was adjusted for savings only for patients that are both treated at our organization and covered by the payor. We assume that the share of cost savings for the integrated organization will reach 50% at year 5 of DPS implementation. Other payors would be the beneficiaries for savings potentially equivalent to the amount we projected, for a total potential savings for the health care organization as a whole of over $12 million. Our organization might ironically suffer decreased revenue from fewer visits and less procedures as a result of more accurate diagnoses. Our environment, a large integrated provider/payor organization is admittedly a special setting. In an environment where the provider (hospital system) is independent from the payor (insurance companies), these calculated cost savings would be accrued completely, and only, by the payors, with no savings accrued by the providers. Again, the providers might shoulder the burden of decreased revenues alone. This also raises the concept that payors might be incentivized to help providers adopt a DPS.

Interpretive errors can occur for several reasons, but naturally there is a spectrum of skill levels amongst pathologists; subspecialty training and experience are contributors to a particular pathologist proficiency level. With an ever-growing knowledge base, the trend in pathology has been toward subspecialization. While many pathologists have mastered one, or a few subspecialties, it is less realistic that one can claim they have mastered all subspecialties. The authors feel that implementation of a DPS will break a longstanding barrier in pathology sharing of cases on a massive scale-and will lead to innovative, more collaborative sign-out models across the pathology practice. The authors envision the above described subspecialization of the entire pathology department as one possibility, whereby nonsubspecialists will develop a niche in two to three subspecialties and over a few years will develop subspecialist level proficiency. While the academic subspecialists might experience increased workload initially due to subspecialist routing, it is expected that as generalists become more subspecialized under the guidance of subspecialists, the subspecialty workload would then redistribute with the gain of their expertise. Digital pathology might allow practices of all sizes and experience levels to materialize. Recruitment would not be limited only to pathologists physically located within that market, but a community hospital normally with enough resources to employ only two pathologists might be able to hire a distributed practice that covers many hospitals and also offers expertise across many subspecialties without needing pathologists to travel to each hospital. Lower volumes might not preclude access to a spectrum of subspecialists. The authors also feel that a DPS could lead to innovations in QA models, whereby more difficult cases can be more quickly referred to more skilled pathologists, who in turn more directly disseminate domain knowledge to pathologists developing proficiency in a subspecialty. A "team sign-out" model might arise when QA is performed prior to sign-out.

We also did not include in our analysis the benefits of any forward-looking features such as the effects of the availability of new tools, including new digital workflow aids or algorithms such as computer-aided diagnosis, on productivity or improvement in diagnostic accuracy. Although these new tools introduced via digital pathology will offer major benefits and enable pathologists to examine tissues in new ways, these effects are largely unquantifiable and inestimable at this time.

While an analysis of the costs of implementing a system is beyond the scope of this study, organizations should be aware that enterprise-wide deployment would require a significant commitment. Deployment would require strategic decisions that take into account the expected usage and scope of implementation, including the number of users, scanners, workstations, and servers. Uptime and redundancy would factor into hardware and software costs. Enterprise software would also require yearly maintenance contracts. Hospital infrastructure would need to be assessed to determine if there was sufficient bandwidth in key locations to support a DPS. Additional on-site support personnel may be required to ensure smooth operation. Operation of the histology laboratory would be deeply affected. Bench space in or near the histology laboratory would need to be committed to. Cutting and staining protocols may need to be adjusted to ensure that only sufficient quality glass slides enter the scanner, in order to attain high quality digital slides. Histotech shifts may need to be adjusted to accommodate for the extra step of scanning slides in order to minimize the effect on turn-around time. Courier routes would need to be adjusted for laboratory consolidation. Vendors cost models are evolving, with some offering up front capital investment costs as well as leasing models based on volume, and others offering combinations of the two.

It also remains to be seen if pathologists can actually save time and be more accurate with digital pathology. We modeled a theoretical 13.4% savings for each pathologist's time, but studies that extensively observe how long pathologists take to examine slides digitally versus on glass are limited. Software design tailored to pathologists needs' will be critical for pathologist adoption of a digital slide workflow. It will be important for software design to accommodate pathologists varied volume, specimen types, and timeliness needs. Good ergonomics, especially for repetitive tasks such as digital slide navigation will help pathologists stay efficient and avoid injury.

There are many pathology practices that employ a subspecialty sign-out model with an efficient courier system, and without a DPS. However, this could not be realistically achieved with the size and scope of our organization, which already utilizes a sophisticated tracking enabled courier system. Our courier system services mainly the subspecialty model of the core academic hospitals, and in the new model, without digital pathology, would require that the courier system distribute nearly all 1.7 million slides to a specific subspecialist. For example, our neuropathologists might be distributed between four sites, and the courier system would need to be aware of which neuropathology cases, regardless of where they originated, would be delivered to which neuropathologist on a case by case basis. Such a courier system would be highly costly, highly complex, and still prone to lost slides. A DPS enables more liquidity of cases, in much the same way that E-mail and text messaging offers much greater liquidity for written communication. Similarly, many multi-site practices have consolidated histology laboratories without DPSs, but our wide geographic distribution and large number of hospitals makes this option less attractive.


   Conclusion Top


Via a comprehensive cost savings analysis for a 5-year roll out of DPS implementation across our health care system we identified two major areas for potential cost savings: Laboratory and pathologist efficiency gains and the ability to render more accurate diagnoses, thereby reducing incremental costs due to interpretive errors for cancer cases. These savings could potentially save the health care organization approximately $18 million over 5 years. As long as the cost of system implementation does not significantly exceed this figure, executive support for adoption of digital pathology within the health care organization may be expected. Despite the limitations of our methods, the proposed cost saving model described hereby may serve as a helpful template for other organizations, allowing them to apply their specific scenarios and in turn estimate their own cost savings for implementation of a DPS implementation.


   Acknowledgement Top


This material was supported by research sponsored by the 711 th Human Performance Wing, Air Force Research Laboratory under agreement number FA8650-11-2-6240. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon.

Conflict of Interest: Jonhan Ho, MD, Anil V. Parwani, MD, PhD, and Jeffrey L. Fine, MD, received research funding from Omnyx, LLC, and have the right to receive proceeds from the sale of certain Omnyx products developed in part under the research funding. Jonhan Ho, MD, Anil V. Parwani, MD, PhD, and Jeffrey L. Fine, MD and Liron Pantanowitz, MD, are employed in part by UPMC, which owns one half of Omnyx, LLC. Jonhan Ho, MD, has a consultant agreement with Omnyx, LLC, through the University of Pittsburgh Department of Dermatology, which does not include this research. Stefan M. Ahlers, Orly Aridor, MSc, and John Kuzmishin are employees of UPMC, which owns one half of Omnyx, LLC. Curtis Stratman, MBA, and Michael Montalto, PhD, are employees of Omnyx, LLC.

 
   References Top

1.Stratman C, Ho J. Digital pathology in the clinical workflow. A Time and Motion Study.[abstract]. San Diego, CA: Pathology Visions; 2010.  Back to cited text no. 1
    
2.Pantanowitz L, Valenstein PN, Evans AJ, Kaplan KJ, Pfeifer JD, Wilbur DC, et al. Review of the current state of whole slide imaging in pathology. J Pathol Inform 2011;2:36.  Back to cited text no. 2
    
3.Raab SS, Grzybicki DM. Quality in cancer diagnosis. CA Cancer J Clin 2010;60:139-65.  Back to cited text no. 3
    
4.van Putten PG, Hol L, van Dekken H, Han van Krieken J, van Ballegooijen M, Kuipers EJ, et al. Inter-observer variation in the histological diagnosis of polyps in colorectal cancer screening. Histopathology 2011;58:974-81.  Back to cited text no. 4
    
5.Eriksson H, Frohm-Nilsson M, Hedblad MA, Hellborg H, Kanter-Lewensohn L, Krawiec K, et al. Interobserver variability of histopathological prognostic parameters in cutaneous malignant melanoma: Impact on patient management. Acta Derm Venereol 2013;93:411-6.  Back to cited text no. 5
    
6.Verkooijen HM, Peterse JL, Schipper ME, Buskens E, Hendriks JH, Pijnappel RM, et al. Interobserver variability between general and expert pathologists during the histopathological assessment of large-core needle and open biopsies of nonpalpable breast lesions. Eur J Cancer 2003;39:2187-91.  Back to cited text no. 6
    
7.Ho J, Parwani AV, Jukic DM, Yagi Y, Anthony L, Gilbertson JR. Use of whole slide imaging in surgical pathology quality assurance: Design and pilot validation studies. Hum Pathol 2006;37:322-31.  Back to cited text no. 7
    
8.Jara-Lazaro AR, Thamboo TP, Teh M, Tan PH. Digital pathology: Exploring its applications in diagnostic surgical pathology practice. Pathology 2010;42:512-8.  Back to cited text no. 8
    
9.Wilbur DC, Madi K, Colvin RB, Duncan LM, Faquin WC, Ferry JA, et al. Whole-slide imaging digital pathology as a platform for teleconsultation: A pilot study using paired subspecialist correlations. Arch Pathol Lab Med 2009;133:1949-53.  Back to cited text no. 9
    
10.Cornish TC, Swapp RE, Kaplan KJ. Whole-slide imaging: Routine pathologic diagnosis. Adv Anat Pathol 2012;19:152-9.  Back to cited text no. 10
    
11.Isaacs M, Lennerz JK, Yates S, Clermont W, Rossi J, Pfeifer JD. Implementation of whole slide imaging in surgical pathology: A value added approach. J Pathol Inform 2011;2:39.  Back to cited text no. 11
[PUBMED]  Medknow Journal  
12.Shah NR, Chokshi DA. Should health care systems become insurers? JAMA 2013;310:1561-2.  Back to cited text no. 12
    
13.Troxel DB. Pitfalls in the diagnosis of malignant melanoma: Findings of a risk management panel study. Am J Surg Pathol 2003;27:1278-83.  Back to cited text no. 13
    
14.US cancer statistics working group. United States cancer statistics: 1999-2010. Incidence and mortality web-based report. 2007 top ten cancers. Atlanta, GAU.S. Department of health and human services, centers for disease control and prevention, and national cancer institute; 2007. Available from: http://www.cdc.gov/uscs [Last accessed on 2014 Mar 10].  Back to cited text no. 14
    
15.Price JA, Grunfeld E, Barnes PJ, Rheaume DE, Rayson D. Inter-institutional pathology consultations for breast cancer: Impact on clinical oncology therapy recommendations. Curr Oncol 2010;17:25-32.  Back to cited text no. 15
    
16.Alexandrescu DT. Melanoma costs: A dynamic model comparing estimated overall costs of various clinical stages. Dermatol Online J 2009;15:1.  Back to cited text no. 16
    
17.National (unadjusted) Medicare reimbursement rates. 2013. Available from: http://www.cms.gov/apps/ama/license.asp?file=/hospitaloutpatientpps/downloads/January-2013-Web-Addendum-A.zip [Last accessed on 2014 Mar 10].  Back to cited text no. 17
    
18.Nitrosi A, Borasi G, Nicoli F, Modigliani G, Botti A, Bertolini M, et al. A filmless radiology department in a full digital regional hospital: Quantitative evaluation of the increased quality and efficiency. J Digit Imaging 2007;20:140-8.  Back to cited text no. 18
    
19.Mackinnon AD, Billington RA, Adam EJ, Dundas DD, Patel U. Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: Results of a 5-year audit. Clin Radiol 2008;63:796-804.  Back to cited text no. 19
    
20.Owens SR, Wiehagen LT, Kelly SM, Piccoli AL, Lassige K, Yousem SA, et al. Initial experience with a novel pre-sign-out quality assurance tool for review of random surgical pathology diagnoses in a subspecialty-based university practice. Am J Surg Pathol 2010;34:1319-23.  Back to cited text no. 20
    



 
 
    Tables

  [Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6], [Table 7], [Table 8]


This article has been cited by
1 Enterprise Implementation of Digital Pathology: Feasibility, Challenges, and Opportunities
DJ Hartman,L Pantanowitz,JS McHugh,AL Piccoli,MJ OLeary,GR Lauro
Journal of Digital Imaging. 2017;
[Pubmed] | [DOI]
2 Digital pathology in clinical use: where are we now and what is holding us back?
Jon Griffin,Darren Treanor
Histopathology. 2017; 70(1): 134
[Pubmed] | [DOI]
3 Validation of whole-slide imaging in the primary diagnosis of liver biopsies in a University Hospital
Adela Saco,Alba Diaz,Monica Hernandez,Daniel Martinez,Carla Montironi,Paola Castillo,Natalia Rakislova,Marta del Pino,Antonio Martinez,Jaume Ordi
Digestive and Liver Disease. 2017;
[Pubmed] | [DOI]
4 Clinical Applications of Whole-slide Imaging in Anatomic Pathology
Zoya Volynskaya,Andrew J. Evans,Sylvia L. Asa
Advances In Anatomic Pathology. 2017; 24(4): 215
[Pubmed] | [DOI]
5 An update on the validation of whole slide imaging systems following FDA approval of a system for a routine pathology diagnostic service in the United States
BF Boyce
Biotechnic & Histochemistry. 2017; 92(6): 381
[Pubmed] | [DOI]
6 Enabling digital pathology in the diagnostic setting: navigating through the implementation journey in an academic medical centre
Chee Leong Cheng,Rafay Azhar,Shi Hui Adeline Sng,Yong Quan Chua,Jacqueline Siok Gek Hwang,Jennifer Poi Fun Chin,Waih Khuen Seah,Janel Chui Ling Loke,Roy Hang Leng Ang,Puay Hoon Tan
Journal of Clinical Pathology. 2016; 69(9): 784
[Pubmed] | [DOI]
7 Overview of Telepathology
Navid Farahani,Liron Pantanowitz
Clinics in Laboratory Medicine. 2016; 36(1): 101
[Pubmed] | [DOI]
8 Applicability of digital analysis and imaging technology in neuropathology assessment
William D. Dunn,Marla Gearing,Yuna Park,Lifan Zhang,John Hanfelt,Jonathan D. Glass,David A. Gutman
Neuropathology. 2015; : n/a
[Pubmed] | [DOI]
9 Overview of Telepathology
Navid Farahani,Liron Pantanowitz
Surgical Pathology Clinics. 2015; 8(2): 223
[Pubmed] | [DOI]



 

 
Top
  

    

 
  Search
 
   Browse articles
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
   Introduction
   Methods
   Results
   Discussion
   Conclusion
   Acknowledgement
    References
    Article Tables

 Article Access Statistics
    Viewed4212    
    Printed81    
    Emailed7    
    PDF Downloaded1059    
    Comments [Add]    
    Cited by others 9    

Recommend this journal