Population-scale genomic programs generate large volumes of variant data that must be processed using customer-configured security and access controls tailored to the intended use, enriched with biological context, organized for cohort analysis, and made accessible to authorized researchers. This reference architecture illustrates how OCI services can be used to transform outputs from existing bioinformatics pipelines into access-controlled, searchable cohort data for research and genomic cohort-analysis workflows.
The architecture illustrates two complementary paths built on a shared genomic data foundation with defined access controls. The first, Genomic Data Ingestion, Annotation, and Indexing, shows how genomic variant data can be ingested using private connectivity and defined access controls, processed and annotated at scale, stored in Apache Iceberg as versioned snapshots, and indexed in OCI Search with OpenSearch for discovery. The second, AI-integrated Genomic Cohort Exploration, shows how authorized researchers can use natural-language questions to initiate authorized search and cohort-analysis operations. The numbered steps in the diagram outline these paths and illustrate how processing, storage, search, analytics, and AI-assisted researcher interaction remain separated. Access and actions are controlled by the application, and researchers review the AI-generated summaries.
Figure 1. OCI reference architecture for AI-integrated genomic cohort analytics
1. Access-Controlled Ingestion and Decoupled Processing
Genomic studies need a reliable way to move existing bioinformatics workflows to the cloud while maintaining data integrity, traceability, and security. Separating data ingestion from downstream processing helps keep data transfers running even if processing systems temporarily fail, reducing the need to transfer study data again.
Genomic variant data and associated study metadata move from the on-premises bioinformatics environment to Oracle Cloud Infrastructure (OCI), over Oracle Cloud Infrastructure FastConnect and land in OCI Object Storage. When the final readiness manifest is written, Object Storage—configured to emit object events—generates an object-create event that OCI Events routes to a function in OCI Functions. The function validates the submission and publishes a processing task to OCI Queue, decoupling ingestion from downstream compute so processing can scale independently and failed processing tasks can be retried independently of the original transfer. Versioned object references, checksums, and stable processing identifiers support traceability throughout the workflow.
2. Scalable Genomic Annotation and Processing
Before cohort-wide analysis, incoming variants need to be processed into a consistent representation and enriched with relevant biological context. At population scale, annotation can be computationally intensive, so the platform must distribute processing across parallel compute resources.
Containerized annotation workers on OCI Kubernetes Engine (OKE) consume queue tasks and run approved workflows using tools such as Ensembl Variant Effect Predictor (VEP) with versioned resources such as ClinVar and gnomAD. OKE supports scheduling, health management, workload isolation controls, and horizontal scaling, allowing studies or processing partitions to run in parallel where configured. On enhanced OKE clusters, workload identity and least-privilege OCI Identity and Access Management (IAM) policies help control access without long-lived credentials in containers. For exceptionally large re-annotation campaigns or specialized workloads with demanding CPU, memory, or storage throughput requirements, OCI High Performance Computing (HPC) can be added as an optional execution tier when customer benchmarking supports its use for the workload.
3. Authoritative Cohort Data and Search-Optimized Genomic Discovery
Researchers need a versioned cohort dataset to support reproducible analysis and an interactive way to discover relevant genomic information. Separating the authoritative analytical layer from the search-optimized layer lets each serve its purpose without requiring one data store to optimize simultaneously for access control, historical reproducibility, and interactive retrieval.
Processed cohort data is stored in OCI Object Storage and organized as Apache Iceberg tables. Apache Polaris, deployed on OKE, provides the Iceberg REST catalog, while OCI Database with PostgreSQL provides persistence for Polaris service state. Apache Spark workloads on OCI Data Flow can be configured to use Apache Polaris as the Iceberg REST catalog to help resolve selected tables and snapshots. The Spark workload then reads the corresponding Iceberg data from OCI Object Storage. OCI Search with OpenSearch provides a derived layer for interactive genomic discovery.
4. Access-Controlled Natural-Language Cohort Exploration
Genomic cohort analysis often requires specialized knowledge of data structures, search tools, and analytical query languages. Natural-language interaction can make the platform easier for authorized researchers to use. In this reference architecture, the AI model should not be granted unrestricted access to genomic data or serve as the system of record for analytical results.
Under the illustrated design, the cohort exploration application helps authenticate the user and uses OCI Generative AI to generate a draft structured data operation based on the user’s request. The application validates the proposed operation against configured allowlists, user entitlements, and query limits. If approved, the application invokes the corresponding backend API through OCI API Gateway. OCI API Gateway and the backend APIs can provide additional authentication and authorization controls for these operations. OCI Generative AI does not directly execute the operation or access the underlying genomic data.
Depending on the validated operation, the application invokes one or both of the following backend APIs: the Variant Search API and the Cohort Analytics API. The Variant Search API queries OCI Search with OpenSearch and returns matching candidates. The Cohort Analytics API submits a bounded Apache Spark workload to OCI Data Flow. For cohort analytics, Spark uses Apache Polaris to help resolve the selected Iceberg table and snapshot and reads the corresponding data from OCI Object Storage to perform the analysis. After the selected operation or operations complete, the application sends only the authorized, structured results to OCI Generative AI to generate a draft summary for researcher review, rather than providing the model unrestricted access to the underlying genomic data.
Validation, application-level guardrails, and researcher review can help reduce unsupported outputs. Data minimization practices can help limit unnecessary exposure of sensitive genomic information. Customers should configure OCI Logging, OCI Monitoring, and OCI Audit in accordance with their security, privacy, and operational requirements, including appropriate handling of sensitive data in application and service logs.
Summary
By separating ingestion, annotation, authoritative storage, search, and AI-supported cohort exploration, the architecture allows each layer to scale according to its workload while supporting defined data-access and processing boundaries. OKE remains the primary platform for distributed annotation, OCI Data Flow provides managed Apache Spark processing for cohort analytics, and OCI HPC can be introduced when specialized high-performance workloads justify additional compute capabilities.
Note: This reference architecture is intended to illustrate research-oriented genomic cohort exploration and analysis. Oracle does not endorse any specific product or solution built using OCI services described on this page. OCI services are general-purpose platforms and are not intended to provide diagnostic or treatment recommendations. Customers are responsible for validating their products and solutions and ensuring compliance with any applicable regulatory requirements. Workloads that process Protected Health Information (PHI) must apply your organization’s specific controls for encryption, access governance, audit logging, and compliance. OCI provides infrastructure capabilities that can support these controls, while operations depend on how you implement and run those governance controls.