Introduction

As we slowly trying to get into normal life due to the recent pandemic (COVID-19) or other health-related conditions, we need to be greatly thankful to healthcare professionals (doctors, nurses, etc.) who put their personal and family time into helping, diagnosing and treating people to their best effort. The pandemic put heavy taxing on healthcare professionals and also brought to light supply-chain issues, and great resignation leading to a shortage in human resources i.e. workforce challenges, to effectively diagnose, treat and address the needs of patients in a timely manner.  Also, the number of healthcare providers in underdeveloped or developed countries saw a surge in demand for health care professionals; to effectively diagnose their patient population to meet ongoing patient needs. This challenges made healthcare providers to look for other alternatives e.g. access to a remote healthcare consultant to help in treating or diagnose patients as needed, to mitigate the on-going and future risk. The pandemic also brought immense workforce challenges to healthcare care organizations; in effectively hiring & training existing or new health care professionals, and scale linearly. As we speak there is not an effective platform exists today, that connects healthcare professionals [or specialties] in digital virtual place and able to diagnose, treat patients and learn collaboratively. This document or article focus on how to build such a high level conceptual solution to mitigate the challenges that discussed so far.

Challenges:

The traditional online, class-based learning methods and remote connected platforms are not effective in the changing world to effectively scale resources to treat & diagnose, train, learn and collaborate from anywhere in the world.  Also, there is no platform which exists for healthcare professionals that enables them to enter into immerse digital space, and can collaborate to diagnose a real-life scenario remotely. What we or healthcare organizations need is a platform that is able to engage, collaborate and immerse the learners into a digital world that will be able to effectively train and diagnose. In order to provide this kind of digital immersive experience, we need to move away from traditional online collaboration tools, class base methods and build a platform that provides aspiring healthcare professionals, doctors, etc. ability to learn real-life scenarios in digital world space. In order to build this kind platform, we need to leverage emerging technologies such as Virtual Reality(VR), Augmented Reality (AR), or Extended Reality (XR) that are proving to be more effective and able to provide a learner or healthcare individual, with greater simulated real-life experience in an immersive digital world.

Please note: At this time Security & Compliance, laws and ethics, social bias in virtual platforms, etc. are not addressed in this document. As well as the pros and cons of using VR headsets e.g. dizzy feeling, and nausea is not addressed here. Also implementing and creating an immersive application suite i.e. VR, AR, MR, XR applications need a huge investment, which is also not addressed in this document; as it depends on the specific needs and approaches towards the solution.

Goal & Business Value Proposition

Creating a digital coaching, learning and virtually enabled collaboration platform for healthcare professionals based on immersive technologies that is:

  • Able to train aspiring healthcare professionals (doctors, nurses, radiologists etc.)  by immersing in digital world and diving deeper into every subject 3D Visually and interactive
  • Connect healthcare specialties in digital spaces and virtually collaborate with other[s] and train effectively
  • Connect healthcare specialties in digital spaces and Virtually collaborate to diagnose and provide treatment to the patient  
  • Integrate and Aggregate different digital assets (EMR systems, clinical systems, Imagining Systems etc. into this digital platform to provide value to the learning system, diagnosis, and effective treatment of  patients
  • Able to immerse and effectively collaborate with digital identities (people in virtual) as present next to each other like a classroom or hospital setting in diagnosis and treatment of the patient.
  •  Able to break the emotional barriers so students can learn, and experience life from a new perspective

 

Notional Architecture & Solution Design – Virtual Reality HealthCare Learning Platform

 

As shown below healthcare aspiring (students) and professionals will use Virtual reality headsets to login into a digital or  Virtual immersive world from anywhere that

 

  • Able to view and learn visually 3D MRI, CT-Scan, X-Rays, etc.  In the virtual world with ability to diagnose information that has been learned using ML/AI Machine learning models.
  • Able to collaborate with other professionals as needed
  • Able to view visually and interact with images for effective learning
  • Able to consult as needed from anywhere in the virtual world and immerse themselves with support, and consult with other professionals to effectively diagnose and treat.

Immersive World

What we need to build this kind of platform

In order to build this kind of Platform we need

  • Imaging systems data i.e. Radiology Images (MRI, CT Scan, X-Rays) that has been diagnosed, annotated
  • Access to centralize VNA (Vendor Neutral Archive) for Radiology Image Assets
  • Access to Clinical Systems (EMR, Imaging, etc.) to get effective diagnosis and treatment information to feed into ML/AI models to train.
  • Access to Cloud Based Infrastructure to effectively store large sets of image data, train & build using Computer Vision, NLP AI/ML algorithms; and build ML models while overlaying diagnosis information on Image with integration with Clinical Systems to provide an effective diagnosis information report.

End-To-End Notional Solution Technology Reference Architecture

Find below the end-to-end conceptual reference Architecture for Virtual Reality 3D Immersive experience for the Healthcare Learning platform.

 

There are two process steps involved to build solution

  • Creating Cloud-Based Radiology Image Content  i.e. Image Lakehouse Refer
    1. Please refer here
  • Creating & Streaming Image content from the cloud to Remote HealthCare Professionals

Find below an end-to-end conceptual architecture in creating, Streaming XR application to end clients. This architecture covers the below processes from high level

    1. 3D deployable virtual assets and Applications
    2. Stream XR content from Cloud into end tethered or non-tethered VR HMD devices/clients for Immersive virtual learning. (E.g. Unity Platform, CloudXR, etc.)

Please note below is the conceptual architecture as of now Oracle Doesn’t have NVidia CloudXR SDK available on their marketplace as we speak (July 19  2022) and other streaming protocols that needed to stream to end HMD devices

As show above

  • The Healthcare  Lakehouse contains Raw 2D / 3D Radiology Image assets that have been archived or streamed from enterprise Modality Systems or PACS/DICOM archives (Refer: How do we ingest and provide a greater value from the radiology images here).
  • The Healthcare Lakehouse Object Storage securely stores 2D, and 3D image assets and VR application build and deployable artifacts. This XR application deployable asset is built with integration with a runtime ML diagnosis prediction model for effective learning and treatment.
  • Healthcare Lakehouse contains Clinical Data Assets that have been collected from other clinical systems (EMR etc.). This clinical information is integrated, aggregated, and co-related along with Image Assets; and overlaid with effective diagnosis and annotation information on the image, which can feed into AI/ML model (using Oracle Data Science Platform)  for effective learning and can help to diagnosis at runtime and provide a valuable prediction for better patient treatment.
  • The Fit-for-purpose database i.e. Oracle Autonomous Database (ADW) is used to capture Image assets metadata to display and correlate information at runtime in an immersive world
  • Oracle Compute instances are used for creating VR applications, build artifacts that will be stored in the Object Storage for XR Streaming. E.g. Tools (Unity Platform Cloud Build/ Compute based Build Server with CI/CD in place).
  • Compute instances containing Real-time engine frameworks e.g. WebGL specification based; to display 3D assets in the end client VR Headsets/browsers for immersive learning.
  • Using NVidia CloudXR technologies to stream VR Application assets to the end user (Doctor, Nurse, or aspiring healthcare professional) for an immersive experience.

 

Future Innovation: Metaverse Based Healthcare University

With emerging technologies such as VR, AR, MR or XR and blockchain based we can build a future Healthcare Metaverse University, where real persons can use their digital identity avatars and can meet, collaborate, present and attend workshops in Virtual University from anywhere. (Digital Lands can be bought using Sandbox, Open Sea etc.). Note this document doesn’t discuss about how we build a healthcare university Metaverse, but provides a ground or foundational idea and framework to steer in that direction.

Definition and Terms:

As per Wikipedia

Augmented Reality (AR): Augmented reality is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.

Virtual Reality (VR): Virtual reality is a simulated experience that can be similar to or completely different from the real world. Applications of virtual reality include entertainment, education, and business.

Mixed Reality (MR): Mixed reality is the merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real-time.

Extended Reality (XR): Extended reality is a term referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. E.g. It includes representative forms such as augmented reality, mixed reality, and virtual reality and the areas interpolated among them.

Extended Reality

 

Digital or Virtual Identity or digital Avatar:  Manifestation of Oneself in the digital world or space of himself from the real-world.

Virtual Identities – These are computational implementation of avatars, accounts and characters that represents Doctors and Medical Students to immerse into the Virtual World to get Virtual Experiences