Introduction
Firecracker microVMs offer fast startup and strong isolation with a minimal device model, ideal for ephemeral, multi-tenant workloads. OCI enables Firecracker on both bare metal and, on select VM shapes, via nested virtualization, giving teams flexibility to choose performance or operational convenience. Our measurements framework shows Firecracker on OCI VMs is a viable option, with predictable trade‑offs versus bare metal.
Isolation with Firecracker
Firecracker is an open source microVM monitor (VMM) designed for secure, lightweight workload isolation. It boots minimal Linux guests quickly and uses a very small device model to reduce the attack surface.
As shown in the architecture, Firecracker achieves this through a multi-layered security model:
1. The Virtualization Barrier: Firecracker runs each guest OS and container workload inside a dedicated microVM, providing hardware-level isolation from other tenants.
2. The Jailer Barrier: To further reduce the attack surface, the Firecracker process itself is wrapped in a “Jailer” that uses Linux primitives like cgroups, namespaces, and seccomp to restrict what the VMM can do on the host.
3. Control and Data Planes:
- RESTful API: A control plane that allows you to configure and manage microVMs (vCPUs, memory, etc.) via a simple API.
- Minimalist Device Model: The data plane provides only the essential virtio devices- Net, Block Storage, and a Metadata Service -keeping the code footprint small and secure.
This architecture specifically solves three common problems:
- Cold-start drag for short-lived services and jobs
- The isolation gap between containers and full VMs.
- Density and cost efficiency for multi-tenant platforms needing fast churn and consistent security boundaries.
How OCI enables Firecracker on VMs and BMs
- OCI bare metal (BM): You control the host OS and have direct KVM access, optimal for maximum throughput, lowest tail latency, and highest microVM density. Network and storage can leverage direct device access (for example, local NVMe and SR‑IOV/VFs where applicable).
- OCI virtual machines (VMs): OCI’s VM platform is KVM‑based. Running Firecracker inside an OCI VM requires nested virtualization so that /dev/kvm is available inside the guest. This capability is available on select VM shapes/tenancies. When present, you can run Firecracker similarly to BM with modest overhead. If /dev/kvm isn’t available on your shape, use BM for Firecracker.
- Security posture (both paths): Combine Firecracker’s minimal device model with Linux hardening (cgroups, namespaces, seccomp, and jailer) and standard OCI network isolation to achieve strong tenant separation with low operational friction.
Virtual Machine vs Bare Metal Performance Results
Same software stack and guest artifacts on both hosts (identical Firecracker build 1vCPU & 5GB of memory, kernel/rootfs, and benchmark versions) and same CPU architecture (VM.Standard.E5.Flex vs BM.Standard.E5). Metrics include CPU/memory, storage and throughput/latency.
Note: These are illustrative results from a single environment, not formal benchmarks or guarantees. Results vary by workload, shape, region, and configuration.
| Metric | VM host, 1 microVM (5 trials) | BM host, 1 microVM (5 trials) | Delta(%) |
| CPU (sysbench cpu, events/s) | 1607.22 | 1663.18 | 3.48% |
| Memory bandwidth – Write (MiB/s, 1 MiB blocks) | 29807.96 | 32305.41 | 8.38% |
| Memory bandwidth – Read (MiB/s, 1 MiB blocks) | 50261.02 | 55764.96 | 10.95% |
| Storage – Sequential prepare write (MiB/s) | 646.99 | 748.72 | 15.72% |
| Storage – Random r/w Read (MiB/s, 16 KiB, sync, fsync/100, 60s) | 143.76 | 257.11 | 78.85% |
| Storage – Random r/w Write (MiB/s, 16 KiB, sync, fsync/100, 60s) | 95.84 | 171.41 | 78.85% |
| Storage – Random r/w p95 latency (ms) | 0.14 | 0.075 | 46.43% |
Observations and guidance
1. What the Results Show: The “Nested” Tax
- CPU & Memory Parity: The ~3% CPU gap is remarkably low, showing that OCI’s E5 Flex shapes handle nested virtualization with minimal compute overhead. Memory bandwidth remains within 11%, which is negligible for most application workloads.
- The Storage Divergence: This is the most critical takeaway. Random I/O on Bare Metal is nearly 1.8x faster, and latency is almost half that of the VM.
- Why? On Bare Metal, Firecracker talks directly to the host’s KVM. In a VM, every I/O request must traverse the guest virtio driver, then the OCI VM’s hypervisor, before hitting the physical hardware.
2. What it means for you
- If your workloads are latency‑sensitive or storage‑heavy, prefer BM. If you value operational simplicity and the deltas above are acceptable, VMs are a viable target – provided /dev/kvm is available via nested virtualization.
3. How to decide quickly:
- Latency/IOPS‑critical path → BM.
- General microservices/ephemeral jobs → VM.
Conclusion
Firecracker on OCI pairs microVM agility with strong isolation. VMs that expose /dev/kvm offer practical performance with modest overhead; bare metal delivers the lowest tail latency and highest throughput. Use VMs for most microservices and fleet scenarios and lean on bare metal when storage intensity or latency SLOs are paramount.