Use Cilium chaining with OKE VCN-native pod networking

Introduction

Kubernetes networking is most useful when it fits both the cloud network and the application operating model. On Oracle Cloud Infrastructure (OCI), Oracle Kubernetes Engine (OKE) supports OCI VCN-native pod networking, where pod IP addresses are allocated from a Virtual Cloud Network (VCN) subnet and are directly routable inside the VCN. This model gives platform teams familiar OCI controls for routing, security rules, and VCN Flow Logs.

At the same time, many Kubernetes teams want Cilium capabilities such as endpoint identity, Kubernetes NetworkPolicy enforcement, Hubble flow visibility, and rich eBPF-based observability. A practical way to combine these capabilities is to keep OCI VCN-native pod networking as the base CNI and run Cilium in CNI chaining mode.

This post walks through a working model for OKE with OCI VCN-native pod networking and Cilium generic-veth chaining. The goal is not to replace OCI pod IPAM. Instead, OCI CNI remains responsible for pod IP allocation and the base network path, while Cilium attaches policy and visibility to the pod networking lifecycle.

Core concepts

OCI VCN-native pod networking assigns pod IP addresses from a private pod subnet in the VCN. This allows pods to be reached through normal VCN routing, peering, and on-premises connectivity patterns when the required route and security rules are in place.

Cilium chaining mode is different from a full Cilium primary CNI deployment. In the chaining model, an existing CNI plugin creates the base pod network interface and address. Cilium then runs as a chained CNI plugin and adds Cilium endpoint identity, policy, and observability around that interface.

For this implementation, the validated posture is:

Base pod networking: OCI VCN-native CNI
Cilium mode:         CNI chaining
Chaining plugin:     generic-veth
Cilium IPAM:         delegated-plugin
Routing mode:        native
CNI exclusive:       false
Kube-proxy:          retained in the conservative model

Prerequisites

Before you run the sample, prepare the following:

1. An OCI tenancy with permissions to create or manage OKE, VCN, subnets, route tables, security lists or NSGs, load balancers, and IAM policies.
2. An OCI compartment and either an existing VCN with endpoint, worker, private pod, and optional load balancer subnets, or permission to create the optional disposable reference network included with the sample.
3. A private pod subnet sized for the expected pod density. In VCN-native mode, pods consume IP addresses from the OCI pod subnet.
4. A workstation or private runner with bash, oci, terraform, kubectl, and jq installed and authenticated. Terraform installs Cilium through the Helm provider, so the orchestration scripts do not require the Helm CLI.
5. Network access from the workstation or runner to the OKE API endpoint after cluster creation. The deployment section generates the oke_cilium_chaining kubeconfig context.
6. OCI network rules that allow worker-to-pod, pod-to-pod, DNS, Kubernetes API, and required OCI service access.
7. DNS and outbound access from worker nodes to the required container registries, or equivalent private mirrors. The sample pulls Cilium images plus test images from external registries.
8. A Cilium version validated with the target OKE and Kubernetes version. The publication baseline in this post uses Kubernetes v1.35.2 and Cilium 1.18.11; select versions currently available and supported in your OCI region rather than copying them blindly.

For the optional Gateway API extension, also validate Gateway API CRDs, Cilium kube-proxy replacement, OCI Load Balancer posture, source IP behavior, and rollback steps in a separate environment before using it in production.

The optional network stack is intended only to make the article reproducible from an empty compartment. Enterprise deployments should normally use an approved landing-zone VCN and keep network ownership outside this sample.

Architecture overview

The architecture uses an OKE enhanced cluster with VCN-native pod networking enabled. The VCN contains an endpoint subnet, a worker subnet, a private pod subnet, and an optional load balancer subnet. The pod subnet is the address pool used by OCI CNI for pod IP allocation.

Figure 1: OCI architecture for OKE VCN-native pod networking with Cilium generic-veth chaining

The numbered flows in Figure 1 show the deployment and validation sequence:

1. Operators use the controlled application path to the optional OCI Load Balancer; private OKE API administration is shown in flow 2.
2. A private runner or CI/CD pipeline uses OCI APIs to apply the Terraform stacks, then uses the private OKE API to apply the CNI chain ConfigMap, Cilium Helm release, and validation workloads.
3. When kubelet creates a pod sandbox, OCI VCN-native CNI allocates the pod IP from the private pod subnet.
4. Cilium runs after OCI CNI in generic-veth chaining mode and creates endpoint identity state without taking over pod IPAM.
5. The validation workloads prove allowed and denied traffic while Cilium enforces Kubernetes NetworkPolicy and Hubble records the verdicts.
6. The optional Gateway API path is a separate extension that requires kube-proxy replacement and its own validation results.

The CNI chain itself is the most important part of the design. Figure 2 separates ownership across the private OKE API, worker nodes, VCN-native pod addresses, and the Cilium policy and observability layer. The chained configuration calls the OCI networking plugins first and Cilium after them.

Figure 2: Baseline CNI ownership, pod IP allocation, Cilium chaining, and validation flow

The baseline architecture keeps kube-proxy in place. That is the conservative starting point for proving OCI VCN-native pod IPAM with Cilium chaining. After validating that baseline, you can evaluate Cilium Gateway API as a separate north-south traffic design. OCI CNI continues to own pod addressing, while the optional Cilium configuration introduces kube-proxy replacement and an Envoy-based Gateway API datapath.

What you build

The sample repository contains Terraform and validation scripts that build and test the pattern end to end. Its README deployment guide is the operational source of truth for the current prerequisites, configuration, deployment, validation, and cleanup commands. Clone the repository and run the remaining commands in this post from the repository root:

git clone https://github.com/oracle-devrel/oke-cilium-chaining-sample.git
cd oke-cilium-chaining-sample
cp envs/oke-vcn-native-cilium-chaining.tfvars.example \
  envs/oke-vcn-native-cilium-chaining.tfvars

Edit envs/oke-vcn-native-cilium-chaining.tfvars with the compartment, VCN, subnet, NSG, compute shape, and region values for your tenancy. Unless a section explicitly says otherwise, keep your shell in the repository root so that the relative paths used by the Terraform and validation commands resolve correctly.

For a disposable from-scratch environment, also copy and edit the optional network configuration. Its VCN and subnet names must match the names in the OKE tfvars file:

cp envs/oke-vcn-native-cilium-network.tfvars.example \
  envs/oke-vcn-native-cilium-network.tfvars

The sample uses staged infrastructure:

stacks/00-network  Optional disposable VCN, subnets, Internet/NAT/Service Gateways, and routes
stacks/01-oke      OKE cluster, node pool, and related security-list resources
stacks/03-cilium   Cilium Helm release after kubeconfig access works
scripts/           Validation scripts for OCI CNI, Cilium, smoke tests, and policy tests

Run the end-to-end workflow

The recommended entry point preserves those stages but orchestrates them as one operation:

scripts/deploy-and-validate.sh

The command above is the enterprise path and expects the VCN and subnets referenced by the OKE tfvars to exist. To create a disposable reference network and run the complete workflow from an empty compartment, use:

scripts/deploy-and-validate.sh --provision-network

Disposable-network mode applies stacks/00-network, creates OKE, automatically attaches the generated role-specific OKE security lists to the sample-owned subnets, installs Cilium, and runs every validation stage. The network Terraform state remains separate so cleanup can prove ownership before deleting it.

Existing-network mode assumes the referenced subnets already have equivalent OKE network rules. The example tfvars enables generated security lists. When those generated lists are required and the repository is authorized to update the existing subnet associations, run:

scripts/deploy-and-validate.sh --attach-security-lists

The generated script requires an explicit APPLY. Without the flag, OKE resource creation can succeed while Kubernetes API access or pod networking remains blocked by the existing subnet rules.

The runner performs the following sequence:

1. When requested, initializes, validates, plans, and applies stacks/00-network to create the disposable VCN, subnets, Internet Gateway, NAT Gateway, Service Gateway, and route tables. Private subnets use the Service Gateway for regional Oracle Services Network traffic and NAT for other outbound endpoints.
2. Initializes, validates, plans, and applies stacks/01-oke to create the OKE enhanced cluster and VCN-native managed node pool.
3. For a sample-owned network, attaches the generated role-specific security lists; existing-network mode changes subnet associations only when explicitly requested.
4. Reads the cluster OCID and OCI region from Terraform outputs, generates kubeconfig, and waits for every worker node to report Ready.
5. Calls scripts/cilium-install-verify.sh, which proves base OCI CNI connectivity, applies the generic-veth chaining configuration, and applies stacks/03-cilium.
6. Runs the cross-node endpoint, ClusterIP, service DNS, Cilium service-table, Hubble flow, and Kubernetes NetworkPolicy proofs, writes complete evidence to validation-results/, and prints one end-to-end summary.

For a private Kubernetes API endpoint, execute the runner from a connected private runner and select the private endpoint:

scripts/deploy-and-validate.sh \
  --kube-endpoint PRIVATE_ENDPOINT

If oke_cilium_chaining is already a kubeconfig context for another API server, choose a unique context name without changing the tfvars cluster key:

scripts/deploy-and-validate.sh \
  --kube-context oke_cilium_chaining_live

A successful run ends with a summary similar to the following:

===== END-TO-END SUMMARY =====
Prerequisite network: PASS (sample owned)
OKE cluster and VCN-native node pool: PASS
Subnet security-list attachment: PASS (sample-owned network)
Kubeconfig and Kubernetes API access: PASS
Worker nodes Ready: PASS
OCI VCN-native cross-node connectivity: PASS
Cilium installation and agent health: PASS
Cilium chained endpoint smoke test: PASS
Cross-node ClusterIP and service DNS connectivity: PASS
Cilium service-table visibility: PASS
Hubble forwarded and policy-denied flow visibility: PASS
Kubernetes NetworkPolicy enforcement: PASS
OVERALL RESULT: PASS
Detailed log: validation-results/deploy-and-validate-<timestamp>.log

The following sections show the same workflow stage by stage so that readers can inspect, rerun, or troubleshoot each layer independently.

The important OKE cluster setting is the pod network type:

cluster = {
  type                 = "ENHANCED_CLUSTER"
  cni_type             = "OCI_VCN_IP_NATIVE"
  pod_cidr             = null
  service_cidr         = "10.97.0.0/16"
  public_endpoint      = true
  image_policy_enabled = false
}

The node pool must include a private pod subnet when the cluster uses OCI VCN-native pod networking:

node_pool_pod_network_option_details {
  cni_type          = "OCI_VCN_IP_NATIVE"
  max_pods_per_node = 31
  pod_subnet_ids    = var.pod_subnet_ids
  pod_nsg_ids       = var.pod_nsg_ids
}

Deploy the OKE cluster

By default, the sample assumes that the target compartment, VCN, endpoint subnet, worker subnet, private pod subnet, and optional load balancer subnet already exist. The stacks/01-oke Terraform stack resolves those existing network resources by OCID or name and creates the following resources:

• An OKE enhanced cluster with OCI_VCN_IP_NATIVE pod networking.
• A managed node pool whose worker nodes use the worker subnet and whose pods use the private pod subnet.
• Optional OKE security lists and generated scripts for attaching those lists to existing subnets.

If you need to prepare the network or want to create the cluster manually instead, see Network Resource Configuration for Cluster Creation and Deployment, Creating an Enhanced Cluster, and Using the OCI VCN-Native Pod Networking CNI plugin.

For an isolated evaluation, --provision-network applies the included stacks/00-network before this OKE stack. Treat that network as disposable reference infrastructure, not as a substitute for an enterprise landing zone.

From the sample repository root, initialize, validate, plan, and apply the OKE stack:

terraform -chdir=stacks/01-oke init
terraform -chdir=stacks/01-oke validate
terraform -chdir=stacks/01-oke plan \
  -var-file=../../envs/oke-vcn-native-cilium-chaining.tfvars \
  -out=tfplan
terraform -chdir=stacks/01-oke apply tfplan

Confirm that Terraform created the cluster and node pool with OCI VCN-native pod networking and resolved the intended subnets:

terraform -chdir=stacks/01-oke output cluster_ids
terraform -chdir=stacks/01-oke output cluster_cni_types
terraform -chdir=stacks/01-oke output node_pool_cni_types
terraform -chdir=stacks/01-oke output resolved_network

When security_lists.enabled = true, Terraform creates the security lists and writes generated/attach-security-lists.sh. Terraform does not silently replace the security lists on existing subnets. Review the generated commands and run the script only if this repository is authorized to update those subnet associations:

terraform -chdir=stacks/01-oke output attach_instructions
sed -n '1,240p' generated/attach-security-lists.sh
generated/attach-security-lists.sh

The generated script preserves existing security-list OCIDs and asks you to type APPLY before changing a subnet. If another Terraform state owns the subnets, coordinate the change to avoid configuration drift.

The OKE stack also snapshots the original subnet security-list associations in Terraform state. Later Terraform refreshes therefore keep generated/detach-security-lists.sh pointed at the pre-attachment state instead of treating the attached lists as the new baseline.

Create the kubeconfig after the OKE control plane is active. Set OCI_REGION to the region in your tfvars file:

export OCI_REGION="us-phoenix-1"
export CLUSTER_ID="$(terraform -chdir=stacks/01-oke output -json cluster_ids | \
  jq -r '.oke_cilium_chaining')"

oci ce cluster create-kubeconfig \
  --cluster-id "$CLUSTER_ID" \
  --file "$HOME/.kube/config" \
  --region "$OCI_REGION" \
  --token-version 2.0.0 \
  --kube-endpoint PUBLIC_ENDPOINT

if [[ "$(kubectl config current-context)" != "oke_cilium_chaining" ]]; then
  kubectl config rename-context \
    "$(kubectl config current-context)" oke_cilium_chaining
fi

kubectl --context oke_cilium_chaining get nodes -o wide

For a private Kubernetes API endpoint, run the command from a connected private runner and use --kube-endpoint PRIVATE_ENDPOINT. See Setting Up Cluster Access for the supported access patterns.

Continue only after the worker nodes report Ready. The next two stages create the chaining ConfigMap and install Cilium.

Configure the CNI chain

Create the CNI chaining ConfigMap before installing Cilium. The ConfigMap defines the plugin chain that runs for pod networking:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-chaining-config
  namespace: kube-system
data:
  cni-config: |-
    {
      "name": "oci",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "cniVersion": "0.3.1",
          "type": "oci-ipvlan",
          "mode": "l2",
          "ipam": {
            "type": "oci-ipam"
          }
        },
        {
          "cniVersion": "0.3.1",
          "type": "oci-ptp",
          "containerInterface": "ptp-veth0",
          "mtu": 9000
        },
        {
          "type": "cilium-cni",
          "chaining-mode": "generic-veth",
          "enable-debug": false,
          "log-file": "/var/run/cilium/cilium-cni.log"
        }
      ]
    }

The sample repository includes a helper:

scripts/apply-oci-cilium-chaining-config.sh oke_cilium_chaining

Helper output:

Applied kube-system/cilium-chaining-config for OCI CNI + Cilium generic-veth chaining.

Install Cilium with Helm

The Cilium values are intentionally conservative for the VCN-native chaining model:

cilium = {
  enabled               = true
  version               = "1.18.11"
  install_mode          = "chaining"
  namespace             = "kube-system"
  repository            = "https://helm.cilium.io/"

  ipam_mode             = "delegated-plugin"
  routing_mode          = "native"
  kube_proxy_replacement = false

  chaining_mode         = "generic-veth"
  cni_custom_conf       = true
  cni_config_map        = "cilium-chaining-config"
  cni_config_map_key    = "cni-config"
  cni_exclusive         = false
  cni_external_routing  = true

  enable_ipv4_masquerade = false
  bpf_masquerade         = false
  socket_lb_enabled      = false

  k8s_network_policy_enabled       = true
  enable_non_default_deny_policies = true
  hubble_enabled                   = true
  hubble_relay_enabled             = true
  hubble_ui_enabled                = true
}

For only the Cilium installation and network-validation workflow after the OKE nodes are Ready, run:

scripts/cilium-install-verify.sh oke_cilium_chaining

The wrapper uses only files included in the sample repository. On its first run it performs the base cross-node OCI CNI test, applies the chaining ConfigMap, installs Cilium through stacks/03-cilium, verifies the OCI CNI and Cilium DaemonSets, tests cross-node pod-IP and ClusterIP Service connectivity, inspects Cilium endpoint and service state, captures Hubble forwarded and denied flows, and runs the Kubernetes NetworkPolicy test. It writes the complete command output to validation-results/cilium-chaining-<timestamp>.log. The top-level deploy-and-validate.sh calls this same wrapper after provisioning OKE and configuring cluster access.

If the kubeconfig context is an alias rather than the cluster key in tfvars, provide both values:

TARGET_CLUSTER_NAME=oke_cilium_chaining \
  scripts/cilium-install-verify.sh oke_cilium_chaining_live

To inspect an existing installation without applying Terraform again, use:

scripts/cilium-install-verify.sh oke_cilium_chaining --verify-only

To run only the Cilium installation stage manually, apply the sample Terraform stack directly:

terraform -chdir=stacks/03-cilium init
terraform -chdir=stacks/03-cilium workspace select oke_cilium_chaining || \
  terraform -chdir=stacks/03-cilium workspace new oke_cilium_chaining

terraform -chdir=stacks/03-cilium apply \
  -var-file=../../envs/oke-vcn-native-cilium-chaining.tfvars \
  -var="target_cluster_name=oke_cilium_chaining"

Validate the cluster

First, confirm that the worker nodes are Ready:

kubectl --context oke_cilium_chaining get nodes -o wide

Validation output from the environment used for this article:

NAME          STATUS   ROLES   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                   CONTAINER-RUNTIME
10.61.4.70    Ready    node    v1.35.2   10.61.4.70    <none>        Oracle Linux Server 8.10   cri-o://1.35.2-13.975e05318cb.el8
10.61.7.184   Ready    node    v1.35.2   10.61.7.184   <none>        Oracle Linux Server 8.10   cri-o://1.35.2-13.975e05318cb.el8

Next, confirm that both OCI VCN-native CNI and Cilium are running. In this model, seeing both DaemonSets is expected:

kubectl --context oke_cilium_chaining -n kube-system get ds cilium vcn-native-ip-cni kube-proxy

Validation output:

NAME                DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
cilium              2         2         2       2            2           kubernetes.io/os=linux
vcn-native-ip-cni   2         2         2       2            2           <none>
kube-proxy          2         2         2       2            2           beta.kubernetes.io/os=linux

Inspect the generated Cilium configuration:

kubectl --context oke_cilium_chaining -n kube-system get cm cilium-config -o yaml | \
  egrep 'cluster-name|ipam|routing-mode|cni-chaining|custom-cni-conf|kube-proxy-replacement|enable-hubble|enable-k8s-networkpolicy'

Validation output:

cluster-name: oke-cilium-chaining
cni-chaining-mode: generic-veth
custom-cni-conf: "true"
enable-hubble: "true"
enable-k8s-networkpolicy: "true"
ipam: delegated-plugin
kube-proxy-replacement: "false"
routing-mode: native

Now check Cilium status:

kubectl --context oke_cilium_chaining -n kube-system exec ds/cilium -- cilium-dbg status --verbose

Key output:

Kubernetes:             Ok   1.35 (v1.35.2) [linux/amd64]
CNI Chaining:           generic-veth
Cilium:                 Ok   1.18.11
Cilium health daemon:   Ok
IPAM:                   IPv4: delegated to plugin
Routing:                Network: Native   Host: Legacy
Masquerading:           Disabled
Hubble:                 Ok
KubeProxyReplacement:   False
BPF Maps:               dynamic sizing: on

Check that Cilium has active service backends:

kubectl --context oke_cilium_chaining -n kube-system exec ds/cilium -- cilium-dbg service list

Validation output:

ID   Frontend                Service Type   Backend
3    10.97.5.5:53/TCP        ClusterIP      1 => 10.61.10.76:53/TCP (active)
                                            2 => 10.61.10.253:53/TCP (active)
6    10.97.0.1:443/TCP       ClusterIP      1 => 10.61.0.3:6443/TCP (active)
8    10.97.139.177:80/TCP    ClusterIP      1 => 10.61.7.184:4244/TCP (active)

Prove the datapath with smoke tests

The validation is intentionally split into four proof areas. Each one demonstrates a different part of the chaining model:

The validation scripts accept the kubeconfig context as the first argument and an optional namespace as the second argument. If you do not pass a namespace, the scripts create a timestamped namespace and keep it for inspection.

This makes the validation stronger than a simple pod ping. The tests cover the base OCI datapath, the Cilium chained endpoint lifecycle, the Kubernetes Service path, Cilium’s service view, Hubble observability, and policy enforcement behavior.

When all stages in the combined wrapper pass, it prints a concise summary suitable for attaching to a review or change record:

===== VALIDATION SUMMARY =====
OCI VCN-native cross-node connectivity: PASS
Cilium DaemonSet and agent health: PASS
Cilium chained endpoint smoke test: PASS
Cross-node ClusterIP and service DNS connectivity: PASS
Cilium service-table visibility: PASS
Hubble forwarded and policy-denied flow visibility: PASS
Kubernetes NetworkPolicy enforcement: PASS
OVERALL RESULT: PASS
Detailed log: validation-results/cilium-chaining-<timestamp>.log

First, validate that the base OCI CNI can move traffic:

scripts/pod-connectivity-test.sh oke_cilium_chaining

The script schedules a client pod and a server pod on different worker nodes and performs an HTTP request from the client to the server pod IP. A passing result means the OKE VCN-native datapath can allocate pod IPs from the pod subnet and route cross-node pod traffic before Cilium policy is introduced.

Representative base-CNI output:

NAME              READY   STATUS    IP             NODE
pod-test-client   1/1     Running   10.61.11.109   10.61.4.70
pod-test-server   1/1     Running   10.61.8.205    10.61.7.184

===== cross-node pod HTTP test =====
POD CONNECTIVITY TEST RESULT: PASS

After Cilium is installed, run the Cilium smoke test:

scripts/cilium-smoke-test.sh oke_cilium_chaining

This test repeats the cross-node HTTP check after Cilium is in the CNI chain, creates a ClusterIP Service for the server, and reaches that Service by both ClusterIP and Kubernetes DNS name. It then queries Cilium endpoint and service state and obtains the forwarded-flow evidence from the Hubble CLI included in the Cilium agent pod.

The baseline deliberately retains kube-proxy and sets kubeProxyReplacement=false. Therefore, the Service test proves that the composed OKE, OCI CNI, kube-proxy, and Cilium deployment provides working Service connectivity. The Cilium service-table check proves that Cilium learned the Service, but it does not claim that Cilium exclusively owns Service forwarding.

The smoke test covers these Cilium-specific signals:

1. Cilium remains healthy after installation in generic-veth chaining mode.
2. The client and server pods are visible in Cilium endpoint state.
3. The endpoints keep VCN-native pod IPs, proving Cilium is not doing pod IPAM.
4. Cross-node HTTP works after Cilium has attached endpoint state.
5. ClusterIP and service-DNS requests reach the backend on the other worker.
6. cilium-dbg service list contains the test Service frontend and backend.
7. Hubble is healthy and reports a forwarded flow involving the smoke client.

Representative truncated output from the enhanced script follows. The full service table and Hubble flow set remain in the timestamped validation log; addresses, pod names, and timestamps vary by run.

NAME                  READY   STATUS    IP             NODE
cilium-smoke-client   1/1     Running   10.61.11.181   10.61.4.70
cilium-smoke-server   1/1     Running   10.61.8.222    10.61.7.184

===== cross-node pod-IP HTTP test =====
NOW: 2026-06-29 15:17:19.363811836 +0000 UTC
CROSS-NODE POD-IP CONNECTIVITY RESULT: PASS

===== cross-node ClusterIP service test =====
NOW: <timestamp>
CLUSTERIP SERVICE CONNECTIVITY RESULT: PASS

===== cross-node service DNS test =====
NOW: <timestamp>
SERVICE DNS CONNECTIVITY RESULT: PASS

===== cilium endpoints for smoke pods =====
552    Disabled   Disabled   132591   k8s:app=cilium-smoke-test   10.61.8.222    ready
2775   Disabled   Disabled   141292   k8s:app=cilium-smoke-test   10.61.11.181   ready

===== Cilium service table for <cluster-ip>:80 =====
ID   Frontend              Service Type   Backend
...  <cluster-ip>:80/TCP    ClusterIP      1 => 10.61.8.222:8080/TCP (active)
CILIUM SERVICE TABLE RESULT: PASS

===== Hubble forwarded flows for smoke client =====
<timestamp>: cilium-smoke-<id>/cilium-smoke-client ->
  cilium-smoke-<id>/cilium-smoke-server:8080 to-endpoint FORWARDED (TCP)
HUBBLE FORWARDED FLOW RESULT: PASS

Namespace kept for inspection: cilium-smoke-1782746218
SMOKE TEST RESULT: PASS

The important signals are:

1. The client and server pods are on different worker nodes.
2. Both pod IPs are from the VCN-native pod address space.
3. The HTTP request succeeds across nodes.
4. Cilium endpoint output includes the smoke-test pods.
5. The same backend is reachable through the ClusterIP and service DNS name.
6. Cilium’s service table contains the test frontend and backend.
7. Hubble records a forwarded flow for the test client.

Together, these signals prove that OCI CNI continues to own pod IP allocation and routing, while Cilium is active as the chained policy and observability layer.

Validate Kubernetes NetworkPolicy

Cilium should see Kubernetes endpoints and enforce Kubernetes NetworkPolicy while OCI CNI remains the base pod networking layer.

Run:

scripts/cilium-policy-test.sh oke_cilium_chaining

The script creates:

policy-server
allowed-client
blocked-client

Then it applies a Kubernetes NetworkPolicy that allows only pods labeled access=allowed to reach the policy-server on TCP port 8080:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-only-approved-client
spec:
  podSelector:
    matchLabels:
      app: policy-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: allowed
      ports:
        - protocol: TCP
          port: 8080

Validation output:

===== allowed client should succeed =====
allowed client: PASS

===== blocked client should fail =====
blocked client: PASS

===== Hubble denied flows for blocked client =====
<timestamp>: cilium-policy-<id>/blocked-client <>
  cilium-policy-<id>/policy-server:8080 Policy denied DROPPED (TCP Flags: SYN)
HUBBLE POLICY DENIAL RESULT: PASS

Namespace kept for inspection: cilium-policy-1782746250
CILIUM POLICY TEST RESULT: PASS

This is the policy proof point. The allowed-client succeeds, the blocked-client fails, Hubble records the denial, and the policy server appears as a Cilium endpoint with ingress policy enabled. The pod IPs remain VCN-native; the enforcement decision comes from Cilium policy state attached to those endpoints.

You can also verify that the policy server is a Cilium endpoint with ingress policy enabled:

kubectl --context oke_cilium_chaining -n kube-system exec ds/cilium -- cilium-dbg endpoint list

Validation output:

ENDPOINT   POLICY (ingress)   POLICY (egress)   LABELS                    IPv4          STATUS
468        Enabled            Disabled          k8s:app=policy-server      10.61.8.32     ready
552        Disabled           Disabled          k8s:app=cilium-smoke-test  10.61.8.222    ready
2775       Disabled           Disabled          k8s:app=cilium-smoke-test  10.61.11.181   ready

How Cilium features are used

The solution uses Cilium features in layers. The first layer is required for the OCI CNI chaining design. The second layer is optional and becomes useful when you want Cilium to own more of the application traffic management path.

Optional extension: Cilium Gateway API

Gateway API gives platform teams ownership of GatewayClass and shared Gateway resources while application teams attach HTTPRoute resources. It can extend this design with Kubernetes-native north-south routing, but it changes the validated baseline in an important way: Cilium Gateway API requires kube-proxy replacement, while the baseline in this article intentionally uses kubeProxyReplacement=false.

Validation boundary: Gateway API was not enabled in the environment that produced the baseline results in this article. Treat this section and the included manifests as an evaluation path, not as validated output. Test it on a separate cluster or through a controlled change to the Terraform state that already owns the Cilium release. Cilium also documents limitations for some advanced Layer 7 capabilities in generic-veth chaining mode, so validate the exact OKE, Kubernetes, OCI CNI, and Cilium versions you plan to run.

The optional evaluation has five steps:

1. Install the Gateway API CRDs supported by the selected Cilium version.
2. Enable Cilium kube-proxy replacement and Gateway API through the existing stacks/03-cilium Terraform state.
3. Create a Gateway with gatewayClassName: cilium and attach an HTTPRoute.
4. Confirm that the resulting OCI Load Balancer or Network Load Balancer posture matches the intended subnet, NSG, and exposure model.
5. Validate routing, CiliumNetworkPolicy behavior, Hubble evidence, and rollback before adoption.

Copy the baseline tfvars file and change these fields inside its existing cilium object:

kube_proxy_replacement = true
extra_values_yaml = <<-YAML
gatewayAPI:
  enabled: true
YAML

Install the matching Gateway API CRDs, then apply the change through the same Terraform workspace that manages Cilium:

GATEWAY_API_VERSION=v1.4.1  # Use the version supported by the Cilium release you validate.

cp envs/oke-vcn-native-cilium-chaining.tfvars \
  envs/oke-vcn-native-cilium-gateway-api.tfvars
# Edit the copied file with the two Cilium settings shown above.

kubectl apply --server-side -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml"

terraform -chdir=stacks/03-cilium apply \
  -var-file=../../envs/oke-vcn-native-cilium-gateway-api.tfvars \
  -var="target_cluster_name=oke_cilium_chaining"

The sample repository includes a small web workload, Gateway, HTTPRoute, and CiliumNetworkPolicy. Deploy the workload and manifests only in the separate evaluation environment:

kubectl create namespace gateway-demo
kubectl label namespace gateway-demo \
  oke-cilium-sample=true oke-cilium-test=gateway-api
kubectl -n gateway-demo create deployment web \
  --image=nginxinc/nginx-unprivileged:1.27-alpine
kubectl -n gateway-demo expose deployment web \
  --port=80 --target-port=8080

kubectl apply -f examples/gateway-api/web-gateway.yaml
kubectl apply -f examples/gateway-api/allow-gateway-to-web.yaml

The policy selects the web endpoints and permits only HTTP GET / from Cilium’s special ingress entity to backend port 8080. That rule protects the second leg of the request, from the Cilium Gateway datapath to the backend pod. A broader default-deny design may also need an explicit rule for the first leg from the external world identity to the ingress identity.

Use the following checks as acceptance criteria. Do not copy expected values from another environment:

This keeps the article’s evidence boundary explicit: the OCI CNI and Cilium chaining baseline is proven by the included smoke tests, while Gateway API remains an optional extension that requires its own results before publication as a validated pattern.

Operational guidance

Use this model when you want OCI-native pod addressing and cloud-network visibility while adding Cilium policy and Hubble observability. The operational boundary is important:

• OCI CNI remains the source of truth for pod IP allocation.
• Pod subnet sizing matters because pods consume VCN IP addresses.
• Security rules must allow the worker, pod, API endpoint, and optional load balancer traffic paths that are used by this deployment.
• Cilium must run with cni.exclusive=false so it does not take over as the only CNI.
• ipam.mode=delegated-plugin tells Cilium that the base CNI handles IPAM.
• Keep kube-proxy in the conservative model unless you explicitly validate kube-proxy replacement for your exact OKE, OCI CNI, and Cilium versions.
• If you enable Cilium Gateway API, treat it as a separate architecture decision because it requires kube-proxy replacement and introduces an Envoy-based ingress datapath.
• For Gateway API, validate Gateway and HTTPRoute status, OCI Load Balancer or Network Load Balancer provisioning, listener security, source IP handling, Hubble flows, and CiliumNetworkPolicy behavior.
• Validate Cilium upgrades together with OKE worker image updates and OCI CNI plugin updates.

Clean up the sample

Use the guarded cleanup orchestrator to preview and then destroy the sample in dependency order:

scripts/destroy-all.sh oke_cilium_chaining --plan-only
scripts/destroy-all.sh oke_cilium_chaining --confirm DESTROY

The script removes validation namespaces, destroys Cilium, deletes the chaining ConfigMap, restores the original subnet security-list associations, and then destroys OKE. It deliberately leaves the prerequisite VCN untouched because the sample normally uses an existing network.

When the deployment used --provision-network, preview and explicitly delete the sample-owned network with the same cleanup entry point:

scripts/destroy-all.sh oke_cilium_chaining \
  --plan-only \
  --destroy-network

scripts/destroy-all.sh oke_cilium_chaining \
  --confirm DESTROY \
  --destroy-network

Network deletion is guarded by Terraform ownership: cleanup refuses to proceed unless stacks/00-network contains state for the VCN.

After deleting a sample-owned disposable network, rerun scripts/deploy-and-validate.sh --provision-network to recreate it. For enterprise mode, provide another compatible existing VCN and update the OKE tfvars before rerunning the deployment.

The orchestrator is reusable across environments: pass the kubeconfig context, set TARGET_CLUSTER_NAME only when that context differs from the cluster key in the tfvars file, and use network stack or tfvars overrides only for advanced layouts.

The following commands show the equivalent manual sequence.

The validation scripts intentionally keep their timestamped namespaces for inspection. They label every test namespace with oke-cilium-sample=true, so remove all sample workloads with:

kubectl --context oke_cilium_chaining delete namespace \
  -l oke-cilium-sample=true

Destroy the Cilium Helm release before destroying the OKE cluster:

terraform -chdir=stacks/03-cilium destroy \
  -var-file=../../envs/oke-vcn-native-cilium-chaining.tfvars \
  -var="target_cluster_name=oke_cilium_chaining"

kubectl --context oke_cilium_chaining -n kube-system delete configmap \
  cilium-chaining-config --ignore-not-found

If you ran generated/attach-security-lists.sh, review and run the generated restore script before destroying the OKE stack. This detaches Terraform-created security lists from the existing subnets:

sed -n '1,240p' generated/detach-security-lists.sh
generated/detach-security-lists.sh

terraform -chdir=stacks/01-oke destroy \
  -var-file=../../envs/oke-vcn-native-cilium-chaining.tfvars

Do not destroy the OKE stack first when its security lists are still attached to existing subnets, because OCI can reject deletion of in-use security lists.

Key benefits

Cloud-native pod addressing
Pods receive IP addresses from OCI VCN subnets, so network teams can reason about pod traffic with familiar VCN constructs.

Kubernetes policy enforcement
Cilium can enforce Kubernetes NetworkPolicy on top of the OCI VCN-native datapath.

Hubble visibility
Hubble gives platform teams flow visibility and endpoint-level context from inside the Kubernetes network layer.

Optional Gateway API evaluation path
After validating the baseline chaining model, teams can separately evaluate Cilium Gateway API for Kubernetes-native HTTP routing and L7 policy.

OCI network-control alignment
VCN subnets, NSGs or security lists, OKE networking, and Cilium/Hubble visibility complement each other rather than replacing one another.

Incremental adoption
This model lets teams keep the OCI-supported VCN-native CNI path while validating Cilium features in a controlled way.

Summary

OKE VCN-native pod networking and Cilium chaining solve different parts of the Kubernetes networking problem. OCI CNI provides pod IP allocation from VCN subnets and keeps pod traffic aligned with OCI routing and security controls. Cilium chaining adds Kubernetes-aware policy, identity, service visibility, and Hubble observability.

The result is a practical design for teams that want OCI-native pod networking without giving up Cilium-based policy and troubleshooting workflows. Start by validating the base OCI CNI, apply the chained CNI ConfigMap, install Cilium with delegated IPAM and native routing, and then prove the result with node readiness, Cilium status, service table, smoke, and NetworkPolicy tests. If you need north-south HTTP routing, add Cilium Gateway API as a separately validated extension and prove it with Gateway status, route status, Hubble flows, and CiliumNetworkPolicy results.

To try the pattern, run the sample’s one-command workflow in an isolated OCI compartment, review the generated PASS or FAIL evidence, and use the guarded cleanup command when validation is complete. After the disposable baseline works, adapt the subnet, NSG, routing, and private-runner choices to your approved OCI landing zone.

References and resources

• Network Resource Configuration for Cluster Creation and Deployment
• Creating an Enhanced Cluster
• Setting Up Cluster Access
• Using the OCI VCN-Native Pod Networking CNI plugin for pod networking
• Cilium Generic Veth Chaining
• Cilium Gateway API Support
• Gateway API getting started
• Cilium Network Policy
• OKE Cilium chaining sample repository
• Sample deployment and validation guide