First Principles: Building an app to pandemic scale

March 14, 2022 | 9 minute read
Shakeeb Rahman
Architect
Text Size 100%:

On March 18, 2020, I received a text message from my colleague to jump on a Zoom call. Now.

In response to the increased uncertainty at the start of the pandemic, Oracle offered the US government a system to help understand the efficacy of early-stage COVID-19 treatments, and we needed to develop and deploy this system in days.

In a world of uncertainty, the best weapon is data. Building world-class data systems that can collect, manage, and analyze data is our core expertise. Our task was to arm decision-makers with the insights to make smart decisions and help eradicate this deadly virus.

We planned to work with health experts who were beginning to understand this pandemic, which meant we needed to act fast and adapt to changing requirements. We also needed to be ready to deploy this system at scale, for millions of users, at a moment's notice. In this time of uncertainty, we were certain that we could deliver the seemingly impossible.
So that evening, we began to assemble a crossfunctional team for Oracle's COVID-19 mission and got to work.

Our efforts eventually became a suite of pandemic response systems that help understand, track, and monitor the spread of COVID-19 and handle the logistics of large-scale vaccination programs across the globe.

This blog post provides a look into the underlying requirements and our approach in rapidly building systems for pandemic scale that enabled the most important data collection effort in modern history.

Oracle’s COVID-19 mission

Oracle's COVID-19 mission addresses the following capabilities:

  • Tracking the efficacy of early-stage COVID-19 treatments
  • Understanding the spread of COVID-19 within regional communities
  • Soliciting volunteers for COVID-19 vaccine clinical trials
  • Managing the clearing house of all vaccination records across the US
  • Enabling federal and pharmacy partners to order vaccines
  • Capturing real-time vaccine safety data from people that have been vaccinated
  • Managing vaccination registration, scheduling, and administrations for a global audience

We worked in close collaboration with several government agencies in the United States, including Health and Human Services (HHS), the National Institutes of Health (NIH), US Centers for Disease Control and Prevention (CDC), and several government agencies across Africa and Australia.

Criteria for success

This unique situation yielded a specific criterion for the success of these systems: Security, scalability, and speed.

Security

While security is of paramount importance in any system, these pandemic response systems required the following special security considerations.

  • If any data breaches occurred, the public loses faith in these systems, and health experts lose the ability to capture and analyze this important data to combat this pandemic.
  • These systems handle personal health information, which requires the highest levels of privacy and security requirements, including strict government controls such as data sovereignty.
  • In the US, these systems adhere to strict FedRAMP and FISMA high requirements, which enforce the strictest level of controls for this critical data. OCI’s US Government Cloud regions are already certified for these requirements.
  • We applied Oracle’s strict internal security processes with external assessments and testing to ensure that the systems met the security demands for the critical nature of this data.

Scalability

  • These systems must handle the large number of users and data to be collected across various stages of the pandemic and must be functional at the national level, capable of servicing millions upon millions of users, and available with little to no downtime, across the globe.
  • These systems are external-facing, with numerous functions ranging from soliciting volunteers for COVID-19 clinical trials, to registration and scheduling of vaccination appointments, to capturing real-time health safety information in the form of surveys. Any performance impact or downtime can severely impact this important and necessary work.
  • The data captured by these systems is entered voluntarily by participants, so any performance impact or downtime means the loss of this vital vaccine safety information, further reducing their participation in one of the most important data collection efforts of our time. Any impact in performance or availability hinders the ability of the health experts to make the correct health policy decisions that can have a global impact.

Speed of delivery

  • Speed of delivery is a top priority because these systems are needed immediately and timelines are determined by external factors, such as vaccine manufacturing and regulations. No other systems experienced this level of pressure for delivery speed.
  • This data collection effort was one of the most timely ever, which meant these systems needed to be in place immediately, or health experts risked not being able to understand data holistically.
  • We had no pre-existing solutions to use. We had to build new systems to solve these new problems.

Our approach

The urgency and volatile landscape of the pandemic required us to act fast and respond to changing requirements at the speed of thought. In this unique situation, we had no time to wait around for a complete functional specification before getting started. We had to get moving right away.

Our low code reference architecture outlines the approach we took, and it enabled us to work faster, measure timelines in days and weeks, not months or years, and rapidly iterate toward a solution.

Our team wasn't concerned with how to build a cloud-scale solution. We already built up the building blocks, tested tools, and competent components as part of Oracle Cloud Infrastructure (OCI) to handle that process for us. Instead, we could direct our attention toward working with stakeholders and ensuring we solve the problem at hand.

Our approach included three key components: Our low-code platform Oracle APEX, Oracle Database on Exadata, and OCI.

Oracle APEX

Because of the urgent nature of these systems, we needed get everyone on the same page as quickly as possible. With APEX, we set the pace by developing apps within days and having near-daily demos and review meetings with stakeholders. At these review meetings, we demoed the applications and collect stakeholder feedback.

On multiple occasions, we were on Zoom conferences with Larry Ellison and high-ranking officials from public health agencies, such as the HHS, NIH, and CDC, where we demoed the system quickly, addressed feedback, and codeveloped the application in real time, enabling us to iterate on the fly and see the result instantaneously.

In a one-hour web conference, we accomplished what would have taken days or weeks with a more traditional development approach. This accomplishment was only possible with a platform like APEX, which eliminates much of the complexity inherent to traditional application development, and instead enabled the team to focus on the problem at hand.

Oracle Database and Exadata

These systems were deployed on Oracle Exadata on OCI to meet the performance needs for an application of pandemic scale. Oracle Database running on Exadata enabled our APEX applications to scale without needing to modify the underlying application codebase. With Exadata, our database can run at peak performance, addressing our second priority (scalability) and setting us up to handle even the most mission-critical workloads.

As APEX runs within Oracle Database, our APEX applications enjoyed zero-latency data access and native access to all Oracle database capabilities, allowing us to run text, spatial, or analytical queries without external dependencies. High-availability features, such as RAC, provided us with an active-active architecture to ensure easy failover to a secondary region if any instance failures happened. Finally, Automatic Workload Repository (AWR) reports and real-time instrumentation enabled us to tune our application queries, discover potential bottlenecks, and observe real-time database performance.

Oracle Cloud Infrastructure

Oracle's next-generation Cloud infrastructure enabled the development and deployment of these systems on a rock-solid foundation that eliminated the complexity of external integrations. It provides all the building blocks necessary to assemble the infrastructure to run these systems, from load balancers, Oracle Identity Cloud service (IDCS), Logging, ONS, Monitoring, Object Store, Email Delivery, and Streaming services to Exadata, Oracle Database, and APEX. We used these prefabricated blocks in OCI because they were made to scale elastically.
Because the nature of sensitive HIPAA data that these systems store, our infrastructure and system components must adhere to strict government FedRAMP high standards. Our Government Cloud enabled us to deploy these systems on infrastructure with FedRAMP high and IL5 PATO authorizations, along with the dozens of certifications and attestations for key security standards across the globe, including SOC, ISO, PCI, and HIPAA. Without this government cloud available to us immediately, we had zero chance of achieving our mission objectives.

Being part of the solution

Within ten days of receiving the first text message, the first pandemic response system was online. However, no one expected to end the pandemic with a single system developed in just ten days. Oracle wanted to be a participant in helping combat this pandemic, and our approach of working collaboratively with stakeholders and iterating fast enabled us to be a part of the solution.

As this first system was nearing our production deployment, we were already in conversation with several government health agencies about the subsequent problems that needed to be solved. Since then, we’ve developed and deployed a suite of pandemic response systems that help us understand the spread of COVID-19 across regional health systems, solicit volunteers for the COVID-19 vaccine clinical trials, understand vaccine safety, and map the spread of new COVID-19 variants.

When it became time to start planning for COVID-19 vaccines, we were already talking with government health agencies, and it was simply a matter of understanding the problem to identify areas where we could help. The ambiguous requirements for these systems touch our entire portfolio.

Oracle’s core expertise is developing data-intensive high-scale systems. To create these pandemic systems, we didn’t need to build APEX from scratch to put these systems together, build a cloud-scale database, or complete a cloud infrastructure platform. Instead, we simply assembled the pieces and followed a pattern that we were confident in delivering.

As a result, we developed systems that provided the clearinghouse for all vaccination records in the United States, enabled federal and pharmacy partners to order vaccines, and monitored and analyzed vaccine safety data from vaccine recipients.

The success of these systems culminated in the development of the Oracle Health Management System (OHMS), which has been deployed within the United States and across the globe to manage complete vaccination programs from Africa to Australia.

Success

Of all our pandemic response systems, V-safe is the most public-facing and perhaps the largest deployment of our low-code reference architecture. Since its deployment in mid-December 2020, when the initial vaccines were made available to healthcare workers, it has enrolled over 10 million verified registrants, provided over 140 million health check-ins, and hosted more than 1.2 million daily active users during peak vaccinations.

This data collection effort is the largest and most important in modern history, and it was a direct result of rapidly iterating with daily collaboration with CDC.

V-safe enabled the CDC to successfully monitor the safety of COVID-19 vaccines in near real time, file VAERS reports by contacting vaccine recipients with severe adverse reactions, and find that COVID-19 vaccines are safe for pregnant women. V-safe is also helping to facilitate more adjunct studies to understand the safety and efficacy of these vaccines.

“The Oracle-developed v-safe system enables millions of patients to voluntarily share their vaccination experience and data without compromising their privacy,” said Oracle Chair and CTO, Larry Ellison.

Conclusion

Oracle COVID-19 mission was OCI’s most impactful solution to date, and we were able to put the youngest cloud to the test by helping solve one of the most important problems of our generation.

Oracle Database isn’t new: Customers have used the proven data platform across the globe in every industry to run mission critical systems for nearly four decades. Oracle APEX isn’t new either: Customers have used the proven low-code platform across the globe in every industry to run mission critical systems for nearly two decades.

OCI is new: The next-generation cloud faced its biggest test yet, proving that it can handle any problem. It has proven itself as a cloud provider used by customers across the globe in every industry to run mission critical systems in the short time it has been on the market.

The combination of these three technologies enabled us to develop COVID-19 pandemic-fighting solutions that are secure, scalable to support hundreds of millions of users, and were developed and deployed at break-neck speed.

References

Click to view more First Principles posts.

Shakeeb Rahman

Architect


Previous Post

Why a symbiotic multicloud ecosystem matters for media and streaming

Michael Ritchie | 3 min read

Next Post


From auto racing to AI, these four leaders bring very different workloads to OCI

Jeffrey Erickson | 5 min read