Laboratory computer performance in a digital pathology environment: Outcomes from a single institution
Mark D Zarella1, Adam Feldscher2 1 Department of Pathology and Laboratory Medicine, Drexel University College of Medicine, Philadelphia, PA, USA 2 Department of Computer Science, Drexel University, Philadelphia, PA, USA
Date of Submission
Date of Acceptance
Date of Web Publication
Correspondence Address: Dr. Mark D Zarella Department of Pathology and Laboratory Medicine, Drexel University College of Medicine, 245 N. 15th Street, Philadelphia, PA 19102 USA
Source of Support: None, Conflict of Interest: None
Background: In an effort to provide improved user experience and system reliability at a moderate cost, our department embarked on targeted upgrades of a total of 87 computers over a period of 3 years. Upgrades came in three forms: (i) replacement of the computer with newer architecture, (ii) replacement of the computer's hard drive with a solid-state drive (SSD), or (iii) replacement of the computer with newer architecture and a SSD. Methods: We measured the impact of each form of upgrade on a set of pathology-relevant tasks that fell into three categories: standard use, whole-slide navigation, and whole-slide analysis. We used time to completion of a task as the primary variable of interest. Results: We found that for most tasks, the SSD upgrade had a greater impact than the upgrade in architecture. This effect was especially prominent for whole-slide viewing, likely due to the way in which most whole-slide viewers cached image tiles. However, other tasks, such as whole-slide image analysis, often relied less on disk input or output and were instead more sensitive to the computer architecture. Conclusions: Based on our experience, we suggest that SSD upgrades are viewed in some settings as a viable alternative to complete computer replacement and recommend that computer replacements in a digital pathology setting are accompanied by an upgrade to SSDs.
Keywords: ImageScope, QuPath, solid-state drive, virtual slides, Windows system image
How to cite this article: Zarella MD, Feldscher A. Laboratory computer performance in a digital pathology environment: Outcomes from a single institution. J Pathol Inform 2018;9:44
Computer replacement is commonplace in medicine and often necessary to support increased computational demands, performance and reliability issues accrued over time, and compatibility with new hardware and software. However, budget restrictions often place a burden on departments to either forego some upgrades and suffer the consequences of reduced efficiency and suboptimal user experience or to shift the burden to other areas which may cause some other aspect of the enterprise to suffer. Most institutions adopt an upgrade strategy that includes complete computer replacement when it is determined that a computer has reached end of life. On occasion, some computers experience some mode of failure before the end of life, and yet others exhibit unsatisfactory performance before the end of the support cycle often leading to premature replacement of the computer. Some institutions, however, elect to repair the computer or to upgrade its components (e.g., add computer memory) to address user concerns about performance.
In the context of digital pathology and whole-slide imaging, in particular, computer performance can have an impact on the user experience and ultimately contribute to the degree to which users are willing to embrace these emerging technologies. An ideal implementation may include upgrades to network infrastructure and data storage to efficiently deliver data from the whole-slide scanner to a server and eventually from the server to the users' computers. One method to optimize the transmission of image data is to split large whole-slide images (often exceeding 1 GB in size) into small manageable tiles stored at multiple resolutions so that the desired field of view and level of magnification can be quickly accessed. This strategy is commonly used to reduce loading times by a factor of 100 or more while increasing file size by only negligible amount. An additional method by which whole-slide viewing is optimized is by caching image regions on the local disk as they are loaded so that they can be quickly accessed when a user revisits a field of view. This results in a net reduction of network utilization and decrease in the time it takes to render an image. Since the process of caching and rendering requires some degree of processing by the local computer, consideration must be made to ensure that the computer meets the technical requirements for proper whole-slide viewing.
In 2015, in conjunction with mandatory institution-wide operating system (OS) upgrades, our department embarked on an upgrade strategy focused on replacing all computer hard disk drives (HDDs) with solid-state drives (SSDs). SSDs are storage devices that, such as HDDs, store the computer's OS, programs, and data. However, SSDs use flash memory instead of moving platters and have been reported to exhibit faster data throughput speeds and markedly lower power consumption.,, We reasoned that this upgrade strategy could confer a number of enhancements to the user experience, including faster whole-slide image rendering due to improved caching performance. We examined the effect of computer upgrades on user experience using direct measurements and modeling to simulate upgrade impact across three different upgrade strategies: (1) complete computer replacement, which we refer to as “architecture upgrade;” (2) hard drive (HDD) replacement with SSDs, which we refer to as “drive upgrade;” and (3) the combination of both architecture upgrade and drive upgrade. In this report, we place particular emphasis on whole-slide image viewing and quantitative image analysis.
We examined a total of 35 computers randomly sampled from the Department of Pathology and Laboratory Medicine at Drexel University College of Medicine. The relevant hardware attributes of all 35 are listed in [Supplementary Table 1 [Additional file 1]]. Of these, two groups of four computers each were initially defined and tested using fresh installations of Microsoft Windows 7 Enterprise and a standardized software installation as specified by the institution's IT department. Notably, the software included Sophos SafeGuard (Sophos, Abingdon, UK) drive encryption software. The first of the initial test groups consisted of four Intel Core 2 Duo processor-based computers with 120 GB SSDs (Samsung EVO 840/850). This group was considered the drive upgrade group. The second test group represented an architecture upgrade group and consisted of four Intel i3 processor-based computers with HDDs installed (7200 RPM, 16 MB cache, SATA drives). This group was later subjected to a drive upgrade by replacing the HDDs with SSDs (Samsung EVO 860), thereby creating a third test group that represented a drive/architecture upgrade group. This paradigm enabled a comparison between architecture and drive upgrades, as well as a measurement of the combination of both upgrade strategies. The testing paradigm and hardware attributes of the initial test groups are depicted in [Table 1].
The remaining 27 computers not in the initial test groups were tested without fresh installations of the OS, often after multiple years of use in the laboratory or administrative offices. The requirements for eligibility in the study were as follows:
Windows 7 Enterprise was installed
The computer was on the institution's domain and connected to the network
Sophos SafeGuard drive encryption and Sophos Antivirus were installed and active
Windows and Sophos updates were current
A minimum of 4GB RAM was installed.
Basic operating system tests
We estimated general loading performance by measuring the time to boot to Windows, the time to log into Windows from the login screen, and the time to log out of Windows. Login time was defined as the interval between entering the domain password and the appearance of the Windows desktop. Logout time was defined as the interval between clicking the “log out” button and the appearance of the login screen. We defined boot time according to the period beginning immediately after the completion of the encryption check and ending with the appearance of the login screen; we added this value to the login time to compute the total boot time. In this way, we did not include BIOS loading, encryption checking, or the time it took to manually enter the username and password. However, the boot and login times we recorded included any domain access attempts necessary to complete login. For the purposes of this test, we disconnected the computer from the network in an attempt to standardize this effect across comparisons. Altogether, we repeated this process three times for each computer under test.
Whole-slide viewing tests
Whole-slide image analysis tests
To measure the performance of whole-slide image analysis, we applied nuclear and cell segmentation to an entire H&E whole-slide image using the cell detection algorithm in QuPath with default settings. This algorithm was capable of performing analysis of regions in parallel using multiple CPU cores. The sample H&E image was acquired at ×20 magnification using an Aperio ScanScope XT whole-slide scanner (Leica Biosystems, Wetzlar, Germany). Image size was 40,008 × 25,683 pixels and contained approximately 75 mm2 of total tissue area comprising approximately 500,000 cell nuclei. We wrote a script that sequentially applied 20 iterations of this algorithm, and we measured the time for each iteration to complete. On a subset of computers, we monitored CPU and drive activity as the computer performed the whole-slide image analysis test.
To account for the observed performance characteristics, we measured drive performance, CPU performance, graphics card performance, and memory performance for each computer in the three test groups using the Windows System Assessment Tool module on the Windows command line. We averaged the performance across 10 iterations for each computer.
We compared the impact of architecture and drive upgrades on common pathology tasks, with an emphasis on the growing area of whole-slide imaging. [Table 1] depicts the three initial testing groups which were used to distinguish the effects of architecture upgrades, drive upgrades, and the potentially synergistic (or redundant) effects of both forms of upgrade. A comparison of the performance of the groups in the first row provided an estimate of the effects of architecture upgrade, while a comparison of groups in the first column provided an estimate of the effects of drive upgrade. When comparing the performance of computers that had undergone a drive upgrade to the performance of the same computers before the upgrade, there was a clear difference in how promptly they responded to commands. Perhaps, the most notable example was booting Windows, which was nearly a factor of 2 faster following drive upgrade (31.5 ± 1.7 s vs. 55.7 ± 11.0 s). We also observed a positive impact in performance following drive upgrade for logging into (33.8 ± 0.7 s vs. 37.2 ± 0.8 s) and out of Windows (11.5 ± 0.5 s vs. 12.4 ± 0.2 s), as well as other common OS tasks such as loading software, copying files, and general navigation.
We examined the performance impact of hardware upgrades to end-users navigating whole-slide images and performing routine image analysis tasks. There was a noticeable difference between the test groups when viewing whole-slide images in Aperio ImageScope, Aperio WebScope (a browser-based viewer and part of the eSlide Manager image management suite), and the QuPath whole-slide image viewer [e.g., [Supplementary Video 2]]. Images often appeared to be slower to render when panning and zooming using the HDD-based group, manifesting as the speed of the transition from a low-resolution “blocky” appearance to a clear depiction of the tissue. Using ImageScope and QuPath, we confirmed this observation in whole-slide images stored on a network drive as well as images stored locally. We developed a model to mimic whole-slide viewing based on the notion that image viewers typically access and render multiple image tiles simultaneously. Examining the cache folder associated with ImageScope revealed that it typically stored 10–100 256 × 256-pixel images for every new region of the slide viewed, subject to tile size, and viewing magnification. During navigation, cached files were typically either read from disk (for examining regions previously visited) or written to disk (for storing new regions), suggesting a potentially disk-intensive process. Our model was designed to simultaneously load and render multiple tiles in the Google Chrome web browser to simulate image caching [Figure 1].
Figure 1: Schematic depicting whole-slide viewing model. (a) A graphical depiction of a single frame of the 64-tile test demonstrates the spatial arrangement of tiles within the browser. A tile was replaced when a new tile was loaded, producing a scintillating pattern when observed in video form. (b) The experimental paradigm consisted of a series of 800–1500 trials (denoted by a single-line segment), each separated by a 30-s interval. Each trial consisted of a total of 10,000 tiles presented x at a time (where x is a value between 1 and 256 in 2N increments). The time it took to render all 10,000 tiles was measured
Computer performance running this model was stratified across the three groups. For single-tile rendering, the architecture/drive upgrade group was significantly faster than the drive upgrade group (309.7 tiles/s vs. 197.3 tiles/s, P < 0.001, t-test) which was in turn significantly faster than the architecture upgrade group (197.3 tiles/s vs. 116.0 tiles/s, P < 0.001, t-test). These results imply a complementary effect of drive upgrade and architecture upgrade on whole-slide viewing performance. However, the drive upgrade alone resulted in nearly twice the performance of the architecture upgrade alone. We examined the relationship between rendering speed and number of tiles simultaneously rendered and found that the performance of the HDD-based group rapidly diverged from the others as more tiles were simultaneously loaded [Figure 2]a. When 64 tiles were simultaneously loaded during each simulated “panning” action, the HDD-based group performed much more poorly in comparison to the same four computers with SSDs (370.6 tiles/s vs. 93.9 tiles/s, P < 0.001, paired t-test). While both SSD-based groups steadily improved performance as more tiles were processed in parallel, the HDD-based computers experienced a decline in performance likely due to the significant caching typically used to support viewing, which was confirmed by the drive usage recorded as computers executed the model [Figure 3], top]. As evident in [Table 2], the groups with SSDs exhibited substantially better general drive performance, potentially explaining the performance differences observed. The slower drive speeds led to a bottleneck which required constant drive access to process cached images.
Figure 2: Whole-slide viewing performance. (a) Rendering performance on the whole-slide viewing model was computed by dividing the number of tiles processed by the total processing time for each trial. Ten trials were conducted for each test condition, and the data are shown according to the standard deviation of the measurements. The number of tiles simultaneously rendered serves as the independent variable. (b) The same test was conducted for 27 computers in the field. The parallel performance ratio was computed as the ratio between 64-tile performance and single-tile performance. The black arrow denotes a drive upgrade computer that underperformed on the single-tile test relative to the rest of the group, and the gray arrow denotes a drive upgrade computer that rendered tiles more quickly in the single-tile test than in parallel
Figure 3: Drive and CPU usage during the whole-slide viewing test. Drive usage (top) was monitored and logged using the Windows Performance Monitor before and during the test. The start of the first trial is denoted by the black arrow. Disk transfer/s represents the number of transfers to and from the hard disk drive or solid-state drive/s. Higher values suggest higher data throughput. CPU usage (bottom) was likewise monitored, and the percentage of processor time represents the amount of CPU usage relative to its capabilities. Gray: sample i3-hard disk drive computer. Black: same computer following drive upgrade. The solid-state drive-based computer was able to complete three trials in the time it took the hard disk drive-based computer to complete one. Note that the hard disk drive-based computer never quite reached 100% CPU usage as disk speed appeared to be the bottleneck
The results from 27 additional computers randomly tested in the department support the notion that drive upgrade status consistently influenced rendering time [Figure 2]b. For all computers but one (denoted by the gray arrow), the SSD upgrade was more effective than the architecture upgrade for single-tile viewing speed. Furthermore, whole-slide viewing performance was improved by loading tiles in parallel for computers that had undergone drive upgrades but, with the exception of two computers (denoted by black arrows), led to poorer performance in computers that had not undergone drive upgrade. Within the two groups, there was no relationship between the number of days active in the field and single-tile (drive upgrade: P = 0.61, architecture upgrade: P = 0.10, Spearman's rank-order test) or parallel performance (drive upgrade: P = 0.48, architecture upgrade: P = 0.81, Spearman's rank-order test), making this factor an unlikely source of the differences we observed. Instead, our results suggest that whole-slide viewing is a drive-intensive process that may benefit more from higher performance drives than from upgrades in architecture.
We also examined the impact of hardware upgrades on routine image analysis as applied to whole-slide images. We tested the ability to detect cells in a sample H and E image based on the watershed segmentation algorithm in QuPath and measured the amount of time required to analyze an entire 75 mm2 piece of tissue scanned at ×20 [Figure 4]a. This type of analysis is commonly used to support stain quantification and computer-aided diagnosis.,,, We found that this procedure was highly CPU intensive [Figure 4]b, and as a result, the performance was primarily influenced by architecture rather than drive speed [Figure 4]c. In a separate test, where we measured general CPU performance on a standard encryption test, we confirmed that the architecture upgrade group exhibited much higher CPU performance capabilities than the drive upgrade group [Table 2], likely contributing to the improved image analysis performance of this group. Drive upgrade status had a negligible impact on image analysis performance.
Figure 4: Whole-slide image analysis test. (a) Sample whole-slide image. Scale bar: 2 mm. Inset: higher-resolution view before (right) and following (left) segmentation. (b) CPU usage during the test, blue: sample i3-hard disk drive computer; red: sample Core 2 Duo-solid-state drive computer. (c) Average time each group needed to segment the whole-slide image in its entirety. Twenty iterations were applied for each computer under test
Our observations support the notion that for whole-slide viewing, an SSD upgrade alone represents a significant upgrade to the user experience. However, this improvement was notably diminished when users ran advanced image processing algorithms from their local machines. In this case, upgraded computer architecture had a more prominent impact than drive performance. These results suggest that department leadership and IT personnel should carefully analyze the intended use of the system before deciding on which upgrade strategy to embark. In environments limited by budgets that allow only some of the department's computers to be upgraded, leaders are encouraged to strategically place those computers in areas that are most likely to reflect their intended use. In a typical pathology department, this likely means that computers that are responsible for whole-slide image viewing or are part of shared workflows that may undergo frequent logins, logouts, and booting are better candidates for drive upgrades. In departments where image processing is ubiquitous, decision-makers may conclude that complete computer replacement is necessary. However, a number of modern image processing frameworks support offloading the computing to a central server. This approach may represent significant cost savings to departments, who may instead opt to invest in server architecture (e.g., additional CPU cores to allocate to the image processing server) while selecting only moderate upgrades to local PCs. We believe that the results of our experience demonstrate distinct modes of operation that support this view: whole-slide viewing is more drive intensive than CPU intensive and whole-slide image analysis is more CPU intensive than drive intensive.
There are some implementations that our study did not directly examine. For instance, we examined performance differences only when the drive on which the OS resides is replaced. This fails to account for computers that contain two drives: an OS drive and a data drive. Since it can be cost prohibitive to replace large HDDs with equally large SSDs, a common configuration is to use an SSD as an OS drive and a HDD for a data drive. In this configuration, we expect that OS-specific operations (e.g., login/logout and program loading) may continue to fully benefit from SSD replacement, but whole-slide viewing of images that may reside on the data drive may not fully achieve the performance gains that we describe. Likewise, for implementations in which the Windows system image files reside on a network drive, the substantial differences we observed between computer groups will likely become less prominent. However, it is important to note that many of the performance gains we describe are due to local caching of tiles from visited regions on the OS drive, which relies on OS drive performance. Therefore, we expect that whole-slide viewing is aided by the SSD upgrade regardless of whether the image file is stored on the OS drive, the data drive, or in a remote networked location.
It is important to note that the computers in our upgraded architecture groups contained second-generation Intel i3 processors, which by no means represent a state-of-the-art CPU. Nevertheless, the CPU tests we report in [Table 2] demonstrate a substantial improvement in raw CPU performance over the much older Core 2 Duo-based computers. We expected such a notable improvement in CPU performance to result in markedly better whole-slide viewing performance, but only moderate improvements were observed. The drive upgrade group, with older CPU architecture, still outperformed the architecture upgrade group, demonstrating that drive upgrades were the more effective upgrade strategy for whole-slide viewing, especially under real-world viewing conditions where multiple image tiles are loaded at once. We believe that selecting an even more modern processor may have resulted in additional performance improvements but still may not have exceeded the performance of the drive upgrade. Indeed, one of the computers that we tested in the field had a fourth-generation Intel i5 processor and did not perform measurably different from the i3 computers in the same category on the whole-slide viewing task.
One of the computers in our drive upgrade group measured slower than the other three computers in the group for the whole-slide viewing model, leading to a high variance of the group as a whole [as evident in [Figure 2]a. We purposefully selected computers for this group with different chipsets to reflect the diversity more likely to be encountered in older computers. Although this computer belonged to the same architecture as the others (Core 2 Duo) and was upgraded with the same SSD, the CPU clock speed of this computer was only 2.00 GHz. As expected, performance of this computer on the CPU and memory speed tests was lower than the others from this group (CPU encryption: 60.3 MB/s vs. 79.0 MB/s; memory: 3658.3 MB/s vs. 5607.9 MB/s), potentially explaining the disparity in performance on the whole-slide viewing task. Despite its poor relative performance, it still loaded single tiles at nearly the same speed as the architecture upgrade computers and exceeded their performance in trials where 64 or more tiles were simultaneously loaded. These results demonstrate that even older architectures can perform at a high level when accompanied by fast disk access. This has important implications for mobile applications which may be based on architectures with slower CPUs.
Although the impact of our study may only be slight in large well-funded academic medical centers, we argue that the results we present have significant implications in environments with more limited resources such as rural clinics and low- and middle-income countries. In these areas, SSD upgrades may be sufficient to achieve a digital pathology solution where previous solutions may have been cost prohibitive. Furthermore, as whole-slide imaging continues to expand beyond pathology departments and into other venues with different priorities (e.g., classrooms), smaller targeted upgrades may represent a solution that achieves the desired goals while operating within the organization's budget. At our institution, we experienced a nearly 10-fold cost reduction for the SSD upgrade approach in place of complete computer replacement. However, additional obstacles beyond raw equipment costs may be present, which can include labor costs for the upgrade (especially in environments without in-house IT support), institutional computer contracts, and a potentially higher likelihood of computer failure encountered when electing not to undergo complete computer replacement.
Zarella MD, Bowman D, Aeffner F, Farahani N, Xthona A, Absar F, et al. A practical guide to whole-slide imaging: A white paper from the digital pathology association. Arch Pathol Lab Med Forthcoming 2018. doi:10.5858/arpa.2018-0343-RA. [In press].
Felthousen M. Reducing costs, improving service, and extending the life of computers with solid-state drives. In: Proceedings of the 40th Annual ACM SIGUCCS Conference on User Services. Memphis, Tennessee, USA: ACM; 2012. p. 199-202.
Zarella MD, Garcia FU, Breen DE. A template matching model for nuclear segmentation in digital images of H&E stained slides. In: Proceedings of the 9th International Conference on Bioinformatics and Biomedical Technology. Lisbon, Portugal: ACM; 2017.
Paulik R, Micsik T, Kiszler G, Kaszál P, Székely J, Paulik N, et al. An optimized image analysis algorithm for detecting nuclear signals in digital whole slides for histopathology. Cytometry A 2017;91:595-608.
Dissecting the business case for adoption and implementation of digital pathology: A white paper from the digital pathology association
Giovanni Lujan, JenniferC Quigley, Douglas Hartman, Anil Parwani, Brian Roehmholdt, BryanVan Meter, Orly Ardon, MatthewG Hanna, Dan Kelly, Chelsea Sowards, Michael Montalto, Marilyn Bui, MarkD Zarella, Gerard Slootweg, JuanAntonio Retamero, MarkC Lloyd, James Madory, Doug Bowman