|SYMPOSIUM – INTERNATIONAL ACADEMY OF DIGITAL PATHOLOGY (IADP)
|J Pathol Inform 2015,
Reimagining the microscope in the 21 st century using the scalable adaptive graphics environment
Victor Mateevitsi1, Tushar Patel2, Jason Leigh3, Bruce Levy2
1 Department of Computer Science (MC 152), University of Illinois at Chicago, IL 60607, USA
2 Department of Pathology (MC 847), University of Illinois at Chicago, IL 60607, USA
3 Laboratory for Advanced Visualizations and Applications, University of Hawai'i at Manoa, 309 Pacific Ocean Sciences and Technology Building (POST), Honolulu 96822, Hawaii, USA
|Date of Submission||03-Mar-2015|
|Date of Acceptance||24-Mar-2015|
|Date of Web Publication||03-Jun-2015|
Department of Pathology (MC 847), University of Illinois at Chicago, IL 60607
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Background: Whole-slide imaging (WSI), while technologically mature, remains in the early adopter phase of the technology adoption lifecycle. One reason for this current situation is that current methods of visualizing and using WSI closely follow long-existing workflows for glass slides. We set out to "reimagine" the digital microscope in the era of cloud computing by combining WSI with the rich collaborative environment of the Scalable Adaptive Graphics Environment (SAGE). SAGE is a cross-platform, open-source visualization and collaboration tool that enables users to access, display and share a variety of data-intensive information, in a variety of resolutions and formats, from multiple sources, on display walls of arbitrary size. Methods: A prototype of a WSI viewer app in the SAGE environment was created. While not full featured, it enabled the testing of our hypothesis that these technologies could be blended together to change the essential nature of how microscopic images are utilized for patient care, medical education, and research. Results: Using the newly created WSI viewer app, demonstration scenarios were created in the patient care and medical education scenarios. This included a live demonstration of a pathology consultation at the International Academy of Digital Pathology meeting in Boston in November 2014. Conclusions: SAGE is well suited to display, manipulate and collaborate using WSIs, along with other images and data, for a variety of purposes. It goes beyond how glass slides and current WSI viewers are being used today, changing the nature of digital pathology in the process. A fully developed WSI viewer app within SAGE has the potential to encourage the wider adoption of WSI throughout pathology.
Keywords: Collaboration, digital pathology, scalable adaptive graphics environment, whole-slide image
|How to cite this article:|
Mateevitsi V, Patel T, Leigh J, Levy B. Reimagining the microscope in the 21 st century using the scalable adaptive graphics environment. J Pathol Inform 2015;6:25
| Background|| |
Whole-slide imaging (WSI) has matured to the point where it is technically feasible to reliably produce high quality digital microscopic images in a timely and efficient manner. This imaging technology is currently being used in a limited fashion for consultations, conferences, medical education and research in the United States and for primary clinical diagnosis in other nations. ,, There are many reasons suggested for the partial adoption of WSI into the practice of pathology. These include technical, workflow, financial, and regulatory issues. ,,,
Most WSI viewing programs, and, therefore, the ways that we are utilizing these digital slides, do not vary significantly from the use of glass slides on a microscope. If digital slides are subject to the same limited uses as glass slides, pathologists will continue to choose to keep their microscopes, and WSI will remain limited to the niche applications in which they are being utilized today. However, if we take advantage of the digital nature of WSI and use them in new and novel ways that were not possible with physical slides, then we will make significant progress to convince pathologists to embrace this technology.
A sophisticated and flexible collaborative environment for WSI is one possible solution to this issue. The need for collaboration in patient care, medical education, and translational research continues to grow. This collaboration can be intradepartmental, interdepartmental within a single institution, involve multiple physical institutions within a single healthcare enterprise, or inter-institutional (local, regional, national and international) across multiple healthcare enterprises. The need for such levels of collaboration is driven by a demand for fast and accurate diagnoses to support patient care in an environment of increasing amounts of knowledge and sub specialization within pathology, the desire for enhanced interdepartmental conferences to help clinicians to understand disease and treatment options for their patients, newer methods to support medical education for the next generation of pathologists and other physicians, and the strong preferences by government and private funding agencies for multiple site collaborations in medical research. ,,
The physical nature of the glass slide makes true collaboration difficult, inefficient and expensive. If a pathologist requires an expert consultant to render an opinion, the glass slide needs to be physically transported to that expert. When a pathologist needs to present at a multidisciplinary conference, a special projector for glass slides is required or the pathologist's time is used to produce static images to incorporate into the presentation. Medical education (at both the undergraduate and graduate levels) and medical research have the same physical limitations with the glass slide as the patient care environment.
There are currently several different options for sharing WSI. These include manual sharing of WSI files via generic online storage or specific software for server or cloud-based storage of WSI, which may or may not have included WSI viewing software. There are also web-based commercial products for storing and sharing WSI. While these tools have made the logistics of sharing microscopic images somewhat easier, it has not yet fundamentally changed the nature of these interactions.
The actual WSI image file can be shared with others. The large size of the WSI files (in the 100s of MB to GB size) exceeds most E-mail attachment limits, which typically run between 10 MB and 50 MB.  These files can be shared through file sharing services such as dropbox and box, where you can purchase ample storage space in the TB range. , However, this method of sharing is a simple digital version of sharing physical glass slides that still require the recipient to have appropriate viewer software without providing any additional opportunities for collaboration.
There are server and cloud-based storage solutions for WSI. These allow any number of images to be annotated and stored for later retrieval, and frequently incorporate viewer software. These solutions have been applied to medical education, where glass slides have been replaced with digital slides that students can view on their computers or other devices that can access the web. Examples of this include PathXL, Digital Slidebox and mScope.  Solutions have also been developed for surgical pathology consultations. Several pathology departments have created websites for physicians seeking consultation to easily load slides and patient information with promise of a quick turn-around time, utilizing a variety of home grown or commercial systems. , While these systems are more automated and efficient when compared to a simple file transfer, they still emulate the same unidirectional flow of information of consultations with glass slides.
All of the WSI scanner manufacturers provide software to enable users to view the files. In addition, there are currently options for vendor-neutral slide viewers, such as Open Slide.  All of these WSI viewers can be shared as a window using web-based collaboration systems such as GoToMeeting, WebEx, Skype and Google Hangouts, and in some cases control of that window can be passed between users. These products are limited in their ability to both enable true multiuser collaboration and handle large volumes of ultra-high resolution WSIs.
We believe that to take maximal advantage of the potential of digital microscopic images, we need to go beyond a simple conversion of the physical workflow into a virtual environment, but need to look at reimagining the "microscope" in the age of cloud computing with a specific emphasis on meaningful and efficient collaboration. While this will not answer all of the issues restricting the broader adoption of WSI, it will provide additional justification and impetus to adopt this evolutionary technology.
The Scalable Adaptive Graphics Environment (SAGE2) was chosen as the collaborative tool for this project. SAGE2 is an open-source collaborative windowing environment that runs in a web browser and the cloud, taking advantage of HTML5 and the high-performance graphics and networking capabilities contained in modern web browsers. As a consequence, to connect to SAGE2 and display the contents of the wall, the user only needs the SAGE2 address URL and a HTML5 compliant web browser. SAGE2 is based on SAGE, which is an open-source tool that enables users to share a variety of data intensive information in a collaborative environment. , SAGE demonstrated that providing collaborators an environment in which they could display, share and interact with high-resolution content from a variety of different formats side-by-side and in real time greatly benefited their work.  SAGE2, through its web browser-based interface, addresses many of the limitations of SAGE and the needs of SAGE users for true integration of multiuser applications and a reduced barrier for entry to enhance real-time distance collaboration. SAGE2 goes beyond the commonly recognized web-based collaboration systems, such as WebEx, GoToMeeting, Google Hangouts and Skype.
SAGE2 is able to process and display a variety of file and media formats, including digital-cinema quality video, high resolution images, high-definition video-teleconferences, presentation slides, documents, spreadsheets and shared computer screens. However, it has not been possible to import, display and collaborate using WSIs in the SAGE or SAGE2 environments. Users could show WSIs through sharing of their computer screen running WSI viewer software, but this restricts the ability of the collaborators to meaningfully interact with the image and limits the displayed resolution and the number of images that can be simultaneously viewed. While designed to work on large tiled high-resolution displays, SAGE2 can also display information on single screen monitors of any size and resolution. Both display environments offer opportunities for the use of WSI collaboration tools.
Our hypothesis is that a WSI viewer app, built for use within the SAGE2 environment, will change how digital WSIs are viewed and used in ways that are not possible with conventional glass slides or current WSI viewing software and web-based collaboration systems. WSI in SAGE2 can enhance collaborations between pathologists regardless of distance, change interactions between pathologists and other medical specialties through truly collaborative multidisciplinary conferences, innovate the methods through which we teach pathology and histology in undergraduate and graduate medical education, and support collaborative multi-institutional research efforts. These new workflows in the use of WSI can then help to justify investment in WSI technology and drive increased utilization of WSI in the practice of pathology, medical education, and medical research.
| Methods|| |
We set out to develop a prototype of a WSI-viewing app for use within the SAGE and SAGE2 collaborative environments. This WSI app would be functionally similar to currently available WSI-viewers but would have limited capabilities consistent with its proof-of-concept mission. Similarly, while the long-term goal is to create a viewer that is agnostic to WSI file format, the prototype would be limited to importing a single WSI file format. We choose the. ndpi file format utilized by the Hamamatsu Corporation as we had access to one of their scanners.
Once there was a functioning prototype, a series of demonstration scenarios would be created to test the ability of the app to function and to evaluate the potential for enhanced collaboration in a variety of simulated situations. Scenarios involving patient care consultation, multidisciplinary conferences, and medical education were created. These cases included glass slides prepared from formalin-fixed paraffin embedded tissue samples that were stained with a variety of chemical and immunohistochemical stains that had protected health information removed. They were scanned at 20X objective magnification using a Hamamatsu Nanozoomer 2.0 HT scanner and NDP. scan software.
These WSI image files would be stored on several desktop and laptop, Internet accessible (wired and wireless) computers that had Google Chrome software installed. Google Chrome was chosen as the browser of choice, as at the time of this study it was the most compatible with the latest HTML5 release that SAGE2 was best optimized for, and SAGE2 was initially optimized for this web browser.
Two separate SAGE2 installations would be utilized for these demonstrations. In the pathology department's Pathology Informatics Learning and Innovations Laboratory is a multi-tiled display consisting of eighteen high-resolution Planar 42" monitors in a six by three array and a specifically configured computer server running the SAGE2 display environment and driving the tiled displays in the openSuse Linux operating system [Figure 1]. In the office of one of the authors of this article is a single 55" LG monitor and a Dell Alienware X51 computer running the SAGE2 software in the openSuse Linux operating system [Figure 2].
|Figure 1: Tiled display with scalable adaptive graphics environment in the UIC pathology informatics learning and innovations laboratory|
Click here to view
|Figure 2: Single screen scalable adaptive graphics environment system in pathologistæs office|
Click here to view
| Results|| |
The SAGE2 user interface (UI)
The SAGE2 UI for the single screen and multi-tiled systems are shown in [Figure 3]. Users can access the UI by visiting the website set up for the specific SAGE2 installation. A representation of the screen fills most of the open web browser window. On the right side of the window, there is a series of UI controls that allow the user to create and control a pointer on the screen, share a computer screen, launch an app (not required for the WSI viewer), browse files, arrange the screen display, manage various settings and provide information/help. Users can also view the actual display on their web browser through the UI controls.
|Figure 3: (a) User interface (UI) for tiled display screen scalable adaptive graphics environment (SAGE2) system in Google Chrome browser window. Each rectangular box in the UI represents two planar displays. (b) User interface for single screen display SAGE 2 system in Google Chrome browser window|
Click here to view
The prototype WSI-viewing app was created as described in the methods section and tested. Users were able to successfully upload. ndpi files to the SAGE2 system by either dragging the file onto a representation of the SAGE2 screen on their web browser, or by utilizing the "media browser" button on the control panel, as shown in [Figure 4]. The WSI image loads and is shown on both the single screen display as well as in the Chrome browser in [Figure 5].
|Figure 4: Adding a WSI file to the scalable adaptive graphics environment system is as simple as dragging the file onto the "screen" in the user interface. Depending on the speed of the connection and the size of the file, it may take seconds to minutes to upload the file|
Click here to view
|Figure 5: The whole-slide image is present in the single screen scalable adaptive graphics environment system. It is being viewed on another computer that is viewing the screen through the user interface over the Internet|
Click here to view
Live Demonstration of a Consultation using Scalable Adaptive Graphics Environment 2
A live demonstration of the SAGE2 WSI viewer was created and shown at the International Academy of Digital Pathology (IADP) meeting in Boston, Massachusetts in November 2014. The demonstration scenario involved a simulated consultation between the presenter at the IADP conference and a "consultant" located in Chicago, Illinois utilizing the single screen SAGE2 system.
The presenter and consultant communicated through a videoconference displayed in the SAGE2 system through its screen sharing capability. A WSI from the case requiring consultation was loaded from the laptop located in Boston. Both the presenter and the consultant created individual pointers and discussed the case in real-time, freely and easily manipulating the WSI through the UI. Additional gross and WSI images were requested by the consultant and loaded by the presenter. A representation of that exchange is shown in [Figure 6].
|Figure 6: This is a recreation of the consultation performed at International Academy of Digital Pathology meeting. Four images from the consultation case (two WSI and two gross jpg) and a shared screen showing an open video image are simultaneously being displayed using the single screen scalable adaptive graphics environment (SAGE2) system. This image is from a computer viewing SAGE2 through the user interface over the Internet. None of the participants in the actual consultation used the SAGE2 screen itself, but connected to it through the Internet|
Click here to view
This entire consultation demonstration was performed using two computers interfacing with the SAGE2 installation through the Internet. The computer in Chicago was connected with wired Gigabit Ethernet and the computer in Boston through a public wireless connection provided by the conference center. The SAGE2 computer served as the host for the consultation without either participant being physically present in the room with that system.
Demonstration of Educational Capabilities of Scalable Adaptive Graphics Environment 2
SAGE and SAGE2 were also used to create some demonstration examples for medical education.
Prior to the development of the WSI viewer, SAGE was assessed in the delivery of educational content to pathology residents utilizing "jpg" images. Using standard presentation software, there are limited options for showing comparisons between similar entities. If one image is shown per presentation slide the images are easier to view, but the presenter needs to flip back and forth between slides. Placing multiple images on a single presentation slide allows side-by-side comparisons at the sacrifice of the visible details of each image. SAGE permits the juxtaposition of images without the sacrifice of image quality, because of support for higher resolution files compared with pdf or PowerPoint presentations.
This was demonstrated through the reconfiguring of a presentation on gunshot wound injuries for a group of pathology residents. A screen showing comparisons between entrance and exit wounds was prepared [Figure 7]. This not only provided the ability for easy comparisons, but also stimulated meaningful discussion between the presenter and pathology residents.
|Figure 7: Education using static jpg images in scalable adaptive graphics environment (SAGE) of gunshot entrance and exit wounds. SAGE enables the presenter to display a variety of different gunshot entrance and exit wounds along with textual information on how to distinguish these injuries. A videoconferencing window is open on the left of the display|
Click here to view
This can be taken to the next level through the use of the WSI viewer. WSI images of differential diagnoses or multiple different images from a single case can be shown side-by-side for comparison and discussion. Since SAGE and SAGE2 allow the simultaneous viewing of different classes of files, the presentation is not limited to one file type. The multiple pointer capabilities permit rich interaction between the presenter and participants [Figure 8].
|Figure 8: Educational capabilities of scalable adaptive graphics environment (SAGE) with WSI. (a) Simultaneous display of WSIs of different types of colon polyps can be used to teach the differences between benign, premalignant and malignant entities. A video of a colonoscopy can provide a clinical reference. (b) SAGE was used to demonstrate the value to the system in the presentation of an autopsy case. Gross images, WSI images and textual data from the autopsy are shown together. Residents rotating at a remote hospital joined the discussion through videoconferencing|
Click here to view
| Conclusions|| |
Whole-slide imaging has come of age technologically yet remains in the early adopter phase of the technology adoption lifecycle. One of the reasons adoption appears frozen in this phase is that current WSI viewers do little more then mimic the use of glass slides. The capability of using digital slides in ways that glass slides can never be used will help drive the broader adoption of WSI.
The SAGE is a tool that has been shown to enable collaborative workflows in academia, research and industry. While well suited for medicine, SAGE had not previously been utilized in pathology primarily due to the lack of a way to easily view WSIs. We set out to "reimagine" the microscope, create a prototype for a SAGE digital microscope, and use it to demonstrate the value of the SAGE environment to leverage WSIs in ways that cannot be accomplished with glass slides.
While we were creating the WSI app, a new web and cloud-based version of SAGE was being developed, increasing the collaborative potential for our WSI viewer. Our demonstrations, including a live "consultation" and educational examples, clearly show that SAGE2 is capable of supporting a sophisticated level of collaboration that is currently unavailable and easily achievable. This has the potential to be a transformative development in many areas of pathology and medicine.
Consultations, which are currently a series of sequential unidirectional exchanges (glass slides or digital images flow one way and the "answer" flows back), can become a free multidirectional exchange of information. A pathologist looking for an immediate consultation is no longer limited to their colleagues down the hall with a double-headed scope. SAGE2 eliminates distance as a limiting factor. Consultations will become much more than tapping an expert for an answer to a perplexing case. Instead, they're educational experiences for the requestor, consultant and any other pathologist who joins the exchange and sharing of knowledge.
Multidisciplinary conferences will not be limited to sequential presenters from different departments switching between different presentations on a single screen. All members of the presenting "team" will present together and share their content simultaneously, enabling easy comparisons between different disciplines, such as pathology and radiology. In addition, the open nature of the SAGE2 environment will enable the "audience" to participate through personal pointers and share content through their electronic devices. This encourages a high level of collaboration between health care providers.
Combining WSI with other visual, auditory and textual information will transform medical education. The systems-based learning method being used in undergraduate medical education that combines information from different areas of medicine to focus on a disease process is well suited for the SAGE environment. Pathology residents and other house staff can also make use of the multi-format, collaborative, side-by-side, real-time capabilities of SAGE to enhance their knowledge in their specialty as well as better understanding the interdisciplinary nature of patient care in the 21 st Century. The ability of students at any level to create their own pointers enhances dialog in the classroom and enables team-based learning.
The lack of geographic limits presents both opportunities and challenges. Persons unable to physically attend a conference or educational session can freely participate in the session and interact in meaningful ways. Busy physicians would be able to "attend" conferences regarding their patients from their private office without having to take the time to travel to a conference room in a hospital across town. Despite these capabilities, it remains challenging to replicate the richness of face-to-face interaction, and remote participation may encourage multi-tasking by participants in these activities.
While the pilot study was successful, there is much work to be done in the future to better define the ability of the SAGE environment for WSI viewing and collaboration. The current viewer needs to be expanded to include the different file formats of WSI to become vendor agnostic, and the ability to display different WSI file formats side-by-side needs to be proven. Other enhancements to the viewer that are anticipated include robust annotation tools and an improved file menu to allow participants to easily build and save screens of images to share. The viewer still needs to be deployed, tested and evaluated in actual patient care, medical education and research situations.
| Acknowledgments|| |
SAGE and SAGE2 are trademarks of the University of Illinois Board of Trustees. This work was supported by a grant from the University of Illinois at Chicago Center for Clinical and Translational Science (Grant Award CCTS0713-07). SAGE2 was funded by National Science Foundation Advanced Cyber Infrastructure Program (Award# ACI 1441963).
| References|| |
Romero Lauro G, Cable W, Lesniak A, Tseytlin E, McHugh J, Parwani A, et al.
Digital pathology consultations-a new era in digital imaging, challenges and practical applications. J Digit Imaging 2013;26:668-77.
Pantanowitz L, Szymas J, Yagi Y, Wilbur D. Whole slide imaging for educational purposes. J Pathol Inform 2012;3:46.
Pantanowitz L, Valenstein PN, Evans AJ, Kaplan KJ, Pfeifer JD, Wilbur DC, et al.
Review of the current state of whole slide imaging in pathology. J Pathol Inform 2011;2:36.
McClintock DS, Lee RE, Gilbertson JR. Using computerized workflow simulations to assess the feasibility of whole slide imaging full adoption in a high-volume histology laboratory. Anal Cell Pathol (Amst) 2012;35:57-64.
Ho J, Ahlers SM, Stratman C, Aridor O, Pantanowitz L, Fine JL, et al.
Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization. J Pathol Inform 2014;5:33.
Pantanowitz L, Sinard JH, Henricks WH, Fatheree LA, Carter AB, Contis L, et al.
Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch Pathol Lab Med 2013;137:1710-22.
Parwani AV, Hassell L, Glassy E, Pantanowitz L. Regulatory barriers surrounding the use of whole slide imaging in the United States of America. J Pathol Inform 2014;5:38.
Clarke JR. The use of collaboration to implement evidence-based safe practices. J Public Health Res 2013;2:e26.
Flatley Brennan P, Valdez R, Alexander G, Arora S, Bernstam EV, Edmunds M, et al.
Patient-centered care, collaboration, communication, and coordination: A report from AMIA′s 2013 Policy Meeting. J Am Med Inform Assoc 2014; 0:1-4.
Yao Q, Chen K, Yao L, Lyu PH, Yang TA, Luo F, et al.
Scientometric trends and knowledge maps of global health systems research. Health Res Policy Syst 2014;12:26.
Al-Janabi S, Huisman A, Van Diest PJ. Digital pathology: Current status and future perspectives. Histopathology 2012;61:1-9.
Goode A, Gilbert B, Harkes J, Jukic D, Satyanarayanan M. OpenSlide: A vendor-neutral software foundation for digital pathology. J Pathol Inform 2013;4:27.
Renambot L, Arun R, Rajvikram S, Byungil J, Krishnaprasad N, Vishwanath V, et al
. SAGE: The Scalable Adaptive Graphics Environment. Proceedings of the WACE; 2004. Available from: http://www.evl.uic.edu/files/pdf/sage_wace2004.pdf
. [Last updated 2004 Sep 24; Last cited on 2015 Feb 05].
Jagodic R, Renambot L, Johnson A, Leigh J, Deshpande S. Enabling multi-use interaction in large high-resolution distributed environments. Future Gener Comput Syst 2011;27:914-23.
Marrinan T, Aurisano J, Nishimoto A, Bharadway K, Mateevitsi V, Renambot L, et al
. SAGE2: A new approach for data intensive collaboration using scalable resolution shared displays. In Proceedings of the 10 th
IEEE International Conference on Collaborative Computering: Networking, Application and Worksharing, Miami, FL. Available from: http://www.evl.uic.edu/files/pdf/collaborate_com2014-camera_ready-2.pdf
. [Last updated 2014 Oct 25; Last cited on 2015 Feb 05].
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7], [Figure 8]