Introduction

Hadoop and Spark are often presented as competing technologies, but they solve different parts of the data-platform problem. Hadoop provides distributed storage, resource management, and a foundation for long-running data services. Spark is a distributed processing engine that can run on Hadoop, Kubernetes, or a managed service and process data from several storage systems.

The hadoop-spark-oci-sample repository turns those choices into two Terraform deployment paths on Oracle Cloud Infrastructure (OCI). The native path combines OCI Big Data Service and OCI Data Flow. The open-source path runs HDFS and Spark on Oracle Kubernetes Engine (OKE). Both can be deployed with the Terraform CLI or through an OCI Resource Manager form.

Hadoop and Spark: Different Jobs, Better Together

What Hadoop Offers

Apache Hadoop is a distributed-data framework. HDFS stores large files across multiple DataNodes, while a NameNode manages filesystem metadata. YARN schedules applications and allocates cluster resources. MapReduce provides Hadoop’s traditional batch-processing model, and a Hadoop distribution can also include services such as Hive, HBase, or Trino.

Hadoop is a good fit when data must remain in a persistent distributed filesystem, several engines need to share a governed cluster, or the workload depends on long-running ecosystem services. The trade-off is an always-on platform that requires capacity planning, security, upgrades, monitoring, and recovery procedures.

What Spark Offers

Apache Spark is a distributed compute engine. A driver plans the work and executors process data partitions in parallel. Spark supports batch processing, DataFrames and SQL, Structured Streaming, and machine-learning workflows through APIs for Python, SQL, Scala, Java, and R.

Spark does not provide durable storage. It reads and writes HDFS, OCI Object Storage, databases, event streams, and other connector-backed systems. Its DAG execution model and optional caching can reduce repeated I/O, especially for iterative workloads, but performance still depends on partitioning, data skew, storage, and executor sizing.

Key Differences

The important point is that Hadoop and Spark can work together. HDFS can store the data, YARN can allocate resources, and Spark can execute the analytics job. Spark can also run without HDFS or YARN by using OCI Data Flow or Kubernetes with Object Storage.

The Solution: Two Deployment Paths on OCI

OCI-Native Managed Services

The native folder can deploy OCI Big Data Service (BDS), OCI Data Flow, or both. BDS provides a managed Hadoop environment with HDFS, YARN, Spark, and profile-dependent ecosystem services. The stack exposes cluster sizing, secure and high-availability options, compute-only workers, and a bootstrap script for configuration changes.

Data Flow provides managed Spark applications without a permanent Spark cluster. The stack defines the application language, script, driver and executor resources, arguments, and Spark properties. Scripts, logs, and results are stored in private Object Storage buckets. An optional warm pool can reduce startup time for repeated jobs, but it also introduces a continuous baseline cost.

• Supporting infrastructure: The stack can create or reuse a VCN, creates the required IAM resources, and can add a private operator VM reached through OCI Bastion.

Open Source on OKE

The opensource folder creates an enhanced OKE cluster and installs the Spark Operator so Spark drivers and executors run as Kubernetes pods. HDFS and OCI Object Storage are independent options. When HDFS is enabled, the stack creates a Kerberos KDC, one NameNode, and configurable DataNodes with OCI Block Volume storage. Object Storage access uses OKE Workload Identity instead of static OCI keys.

This path offers more control over images, versions, manifests, and Kubernetes integration, but it is not a complete Hadoop distribution: it does not deploy YARN, MapReduce services, Hive, HBase, or Ranger. The platform team owns the in-cluster data services, security hardening, upgrades, monitoring, backup, and recovery.

• Important scope: The current OKE design is a reference implementation with one NameNode and one KDC, so it is not highly available without further engineering.

How It Works

Native Workflow

Terraform provisions networking, Object Storage, IAM, and the selected managed services. BDS keeps the Hadoop cluster available for HDFS/YARN workloads. Data Flow creates reusable Spark application definitions; a run is submitted when work is required. The two services can be used independently or together, but the sample does not create an automatic pipeline between them.

The included native examples demonstrate serverless CSV-to-Parquet ETL, Spark on YARN with HDFS, repeated Data Flow jobs with an optional warm pool, and a secure/high-availability BDS readiness check.

OKE Workflow

Terraform creates the OKE infrastructure and a private operator VM. The operator then applies the Kubernetes manifests and installs the Spark Operator from inside the VCN. This step is asynchronous, so a successful Terraform Apply does not mean HDFS and Spark are ready; cloud-init and pod status must be checked afterwards.

The OKE examples cover Spark-only processing, Kerberos-authenticated HDFS with Spark, and Spark access to OCI Object Storage. The Object Storage example needs a compatible OCI connector and per-job Spark configuration.

Choosing the Right Path

Use Data Flow for Elastic Spark

Choose Data Flow for intermittent or scheduled ETL, SQL, aggregation, and data-preparation jobs over Object Storage when maintaining a permanent Spark cluster is unnecessary.

Use BDS for a Persistent Hadoop Platform

Choose BDS when workloads require HDFS, YARN, long-running Hadoop services, a managed cluster operating model, or a migration path for existing Hadoop applications.

Use Both for Mixed Workloads

Use BDS and Data Flow together when durable Hadoop services must remain online but bursty or independent Spark jobs benefit from elastic managed compute.

Use OKE When Control Is the Requirement

Choose the OKE path when Kubernetes standardisation, custom images, or component-level control justifies the additional platform-engineering responsibility.

Security and Production Readiness

Both paths include useful security building blocks, but neither removes the need for a production review. Before deployment, confirm the following:

• Identity and state: Protect Terraform state because administrator and Kerberos credentials can be stored there, and review IAM policies for least privilege.
• Networking: Restrict administrative CIDRs, keep workers private, and validate all required service and connector paths.
• OKE policy enforcement: The default Flannel CNI does not enforce the supplied NetworkPolicies; install and test a compatible policy engine before relying on them.
• Availability: Use BDS high availability where required. Add NameNode and KDC resilience, anti-affinity, backups, and restore testing to the OKE design.
• Versions and images: Confirm service and Kubernetes versions in the target region and replace public default images with reviewed, pinned images for production.
• Cost and cleanup: Account for persistent BDS or OKE nodes, pools, storage, gateways, and operator VMs, and test cleanup before using real data.

Use Cases

• Serverless data engineering: Transform raw files in Object Storage with Data Flow and write curated Parquet datasets without operating a Spark cluster.
• Hadoop analytics: Run Spark on YARN against HDFS when data locality or Hadoop services are required.
• Frequent Spark jobs: Use a Data Flow pool when repeated runs need lower startup latency and the baseline capacity is justified.
• Secure managed platform: Use the BDS secure/high-availability profile as a starting point for persistent enterprise workloads.
• Kubernetes data platform: Run Spark on OKE with optional HDFS or Object Storage when container-level control is important.

Conclusion

Hadoop and Spark are not direct substitutes. Hadoop provides durable distributed storage, resource management, and a broader service platform. Spark provides the processing engine and can run on Hadoop, Kubernetes, or a managed service. The repository reflects that relationship through an OCI-native path for managed Hadoop and elastic Spark, and an OKE path for teams that want greater control.

Start with one workload in a non-production compartment. Choose Data Flow for intermittent Spark over Object Storage, BDS for a persistent HDFS/YARN environment, or OKE when Kubernetes control is a real requirement. Deploy the matching folder, run the closest example, validate security, resilience, performance, and cost, and then harden the design before production.