X

A Bridge to the Cloud...

How to Deploy Oracle Linux KVM on Oracle Cloud Infrastructure

Simon Coter
Senior Manager, Oracle Linux and Virtualization Product Management

This document is for test and educational purposes only.

This document is still under review; sections of this document could change and further enhancements and/or options can be introduced on the same.

Introduction

The goal ­of this document it to offer a solution to deploy Oracle Linux KVM (OL-KVM) and Oracle Linux Virtualization Manager (OLVM) 4.3 release on Oracle Cloud Infrastructure.

Assumptions

·      OL-KVM hosts can run only on BM.DenseIO2.52 and BM.Standard2.52 shapes (BM.Standard.E2.64 not tested actually)

·      OLVM host can run on VM.Standard2.2 shape or higher

·      OL-KVM Virtual Machines can only rely on the 2nd physical NIC of the Bare-Metal server (for a total of 26 vNICs dedicated to OL-KVM Virtual Machines running)

·      Live-Migration is not available on this configuration (OL-KVM / OLVM running on OCI)

·      The OLVM Datacenter has to be configured for "Shared Storage"

Networking configuration: VCN and Subnet(s)

The setup requires proper VCN / Subnet(s) configuration; the architecture requires:

·       one unique VCN, where to setup the entire networking configuration for OLVM / OL-KVM and Virtual machines will run on top.

·       Internet access for OLVM and OL-KVM(s) instances 

On this example, within the OCI VCN (10.0.0.0/16) the subnet(s) configured are:

Name

CIDR Block

Subnet Access

Usage

OCI

10.0.0.0/24

Private (Regional)

Dedicated to OCI (Storage and Services)

OLVM

10.0.1.0/24

Public (Regional)

Dedicated to OLVM Service Access, OLVM/KVM Server SSH Access

KVM-VM

10.0.2.0/24

Public (Regional)

Dedicated to KVM Virtual Machine vNICs

Screen Shot 2020-04-08 at 11.23.00 PM.png

Architecture

olvm-kvm-oci.png

Oracle Linux Virtualization Manager deployment requirements

Oracle Linux Virtualization Manager, built on OL7 image (latest available), has following requirements:

Built and created from latest OL7.7 (or higher but lower than OL8) image

·       vNIC (1): associated to "OLVM" Public subnet => Public IP address enabled

·       vNIC (2): associated to "OCI" Private subnet => No Public IP address

Screen Shot 2020-04-08 at 11.28.02 PM.png
Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

·       vNIC(1): olvm (Public IP address available)

·       vNIC(2): oci-olvm

Note: for vNIC(1), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

 

Oracle Linux KVM Server deployment requirements

Oracle Linux KVM Server, built on OL7 image (latest available), has following requirements:

Built and created from latest OL7.7 (or higher but lower than OL8) image

·       vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IP Address

·       vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

kvm-vnic-01.png

Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

·       vNIC(1): olkvm01

·       vNIC(2): vdsm01 (Public IP address available)

Note: for vNIC(2), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

 

Oracle Linux Virtualization Manager installation

By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM.

So, the first step requires to disable the non-required Yum-channels on the OLVM instance by executing the following command:

# yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

 

·       Install and enable required yum repositories

# yum install ovirt-release-el7 -y

# yum-config-manager --disable ovirt-4.2

# yum-config-manager --disable ovirt-4.2-extra

 

·       Take "SELinux" to "permissive" level:

sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

setenforce 0

 

·       Update the system to the latest set-of-packages available

# yum update -y

 

·       Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

[root@ol7-olvm network-scripts]# cat ifcfg-ens5

DEVICE=ens5

ONBOOT=yes

IPADDR=10.0.1.2

NETMASK=255.255.255.0

BOOTPROTO=none

HWADDR=00:00:17:01:AB:EC

MTU=1500

DEFROUTE=no

NM_CONTROLLED=no

IPV6INIT=no

DNS1=169.254.169.254

 

·       Enable the IP address for the second device, dedicated to VDSM; example:

# ifdown ens5

# ifup ens5

 

·       Install "Oracle Linux Virtualization Manager" by executing following command

# yum install ovirt-engine -y

 

·       Generate "ssh-keys" for your OLVM Instance; this step is required to then get access to the KVM-Server on the first boot

[opc@ca-ovsx51 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/opc/.ssh/id_rsa):

Created directory '/home/opc/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/opc/.ssh/id_rsa.

Your public key has been saved in /home/opc/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:Xg17qxVZczew9OMrlFskXIZv74IsNT8SAM8UMPEd0Q8 opc@ca-ovsx51.us.oracle.com

The key's randomart image is:

+---[RSA 2048]----+

|        +o..o=.o |

|        .o..ooE  |

|         =o .++Bo|

|          ++ o===|

|        S o.=o.o.|

|       . . .=oo o|

|        .  ooB o |

|          .o+ = .|

|          .. . o |

+----[SHA256]-----+

NB: the KVM Server instance will boot on a "Private Subnet" (OCI) and this one will be only accessible from IPs/devices running on the same VCN.

 

·       Save the content of your public-key, required while deploying the OL-KVM Bare-metal instance

[opc@ca-ovsx51 ~]$ cat .ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCscZTV40F5st6G/snruzWVuoYmdqxzDhNvtJI3TtNTuHD2UbNIzSGREVH2fjZv7ANwA

mu8Vx7en+rVEUjVByUhQm1vHq9cGbDoia541Cau0qtUp2ZaN54oVQpl4Utz4JLRAsJel03PC0DHyKUmM/uj0CwM2a0Kz9OnCuAmcZ2ttC

egUC9FpSj3WvmtNB3Ca/1kyNFyUmil9J7r3Rc/nbXtydoPVJrd/zec5gwIIn/cDxFOvaoIYGCD0Yshb0Fih8b9VOKWfTQaEuzuwx9BBzu

7NIQ7jtqeYlDwpdVqIFvmiA5XtaMuIfm2+BFbSN4ZFoDeIqFoBfbDKQBTtv5Fb0K5 opc@ca-ovsx51

 

·       Proceed to the "Oracle Linux Virtualization Manager" configuration by following the official Oracle Documentation, available at this link

 

Oracle Linux Virtualization Manager OCI customization

On OCI the "ovirt-engine" service listens on the private-IP address while the service is accessed by the OCI Public IP address.

To properly get the OLVM web interface available following configuration is required:

·       Edit the file "/etc/ovirt-engine/engine.conf.d/11-setup-sso.conf" and change the "SSO_ALTERNATE_ENGINE_FQDNS" with proper FQDN of your OLVM (you can get it from OCI console); example:

[root@ol7-olvm ~]# cat /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf

ENGINE_SSO_CLIENT_ID="ovirt-engine-core"

ENGINE_SSO_CLIENT_SECRET="SHy8iaClAv0avdJveqPwroVaxE51Bast"

ENGINE_SSO_AUTH_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"

ENGINE_SSO_SERVICE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"

ENGINE_SSO_SERVICE_SSL_VERIFY_HOST=false

ENGINE_SSO_SERVICE_SSL_VERIFY_CHAIN=true

SSO_ALTERNATE_ENGINE_FQDNS="ol7-olvm.olvregional.olvvcn.oraclevcn.com"

SSO_ENGINE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/"

NB: Consider that this FQDN will have to be resolved by your client(s) accessing the web-interface (by proper DNS or hosts file).

 

Oracle Linux KVM Server installation

Note: Respect following vNIC configuration for your OL-KVM Bare-Metal Instance:

·       vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IPAddress

·       vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

 

·       Get access to the OL-KVM Bare-Metal Instance by OLVM Instance

client ==> ssh opc@<olvm> ==> olvm ==> ssh opc@<kvm>

 

By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM

·       Disable the non-required Yum-channels on the OLVM instance by executing the following command

# yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

# yum install ovirt-release-el7 -y

 

·       Take "SELinux" to "permissive" level

sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

setenforce 0

 

·       Update the system to the latest set-of-packages available

# yum update -y

 

·       Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

[root@ol7-olvm network-scripts]# cat ifcfg-eno3d1

DEVICE=eno3d1

ONBOOT=yes

IPADDR=10.0.1.3

NETMASK=255.255.255.0

BOOTPROTO=none

HWADDR=00:10:e0:ec:e4:69

MTU=1500

DEFROUTE=no

NM_CONTROLLED=no

IPV6INIT=no

DNS1=169.254.169.254

 

·       Enable the IP address for the second device, dedicated to VDSM; example:

# ifdown eno3d1

# ifup eno3d1

 

·       Due to compatibility issues between "Ksplice" and the OVA import process managed by OLVM, uninstall Ksplice

# yum remove ksplice* -y

# rm -f /sbin/modprobe

# mv /sbin/modprobe.ksplice-orig /sbin/modprobe

 

·       Proceed to the OL-KVM host configuration by following the official Oracle Documentation, available at this link.

Note: the discover will have to happen on the dedicated "VDSM" subnet (2nd physical NIC of the Bare-Metal Instance)

 

Oracle Linux KVM Server OCI customization

This chapter is dedicated to the required customization to get an OCI OL-KVM Bare-Metal Instance manageable by Oracle Linux Virtualization Manager.

The required customization is related to the NIC(s) and Virtual-Function(s) management; on OCI, for each reboot of the BM Instance, the Virtual-Functions change their HW-ADDR (or Mac addressed).

Due to this important change, we need to instruct OLVM, with proper updates.

Following steps will show how-to create one new Linux service, dedicated to this target:

·       Open "/etc/default/grub" and add the following line to the end of the "GRUB_CMDLINE_LINUX" entry

Intel_iommu=on

 

The file will look as follows:

GRUB_CMDLINE_LINUX="crashkernel=auto LANG=en_US.UTF-8 console=tty0 console=ttyS0,9600 rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ip=dhcp netroot=iscsi:169.254.0.2::::iqn.2015-02.oracle.boot:uefi iscsi_param=node.session.timeo.replacement_timeout=6000 intel_iommu=on"

 

·       Enable "tuned" and set the performance optimization for "virtual-host"

# systemctl enable tuned

# systemctl start tuned

# tuned-adm profile virtual-host

 

·       Commit the changes so that this configuration will always be used at boot time

cp /boot/efi/EFI/redhat/grub.cfg /boot/efi/EFI/redhat/grub.cfg.orig

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

 

Get OLVM aware of NIC "Virtual Function(s)" now enabled on the OL-KVM Server.

·       Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host

net-00.png

·       Click on "Network Interfaces" tab and click on "Setup Host Networks" button.

net-01.png

·       Click on the "Edit" icon of the "2nd Physical NIC" to manage the SR-IOV NIC options.

net-03.png

·       On the "Edit Virtual Functions (SR-IOV) configuration of <NIC>" window, expand the "Number of VFs setting" option; then write "26", the number of Virtual Function(s) today supported on OCI BM Shapes.

net-04.png

·       Confirm with "OK" and then, again, click on "OK" button to proceed to the Virtual Functions discovery and enablement process. Those OCI Virtual Functions will be then leveraged by KVM Virtual Machines for networking.

 

DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

  • Reboot your "Oracle Linux KVM server" by leveraging OCI web management interface to boot with updated system.

 

Oracle Linux KVM Server: create and define Storage Domain(s)

One requirement to get your OLVM Datacenter up (and, so, enabled) is to configure, at least, one storage domain for your Datacenter.

To accomplish this step you can use local NVME storage (on BM.DenseIO2.52 shape) or OCI Block-Volumes (on BM.DenseIO2.52 and BM.Standard2.52 shapes).

For details on how-to get the storage correctly configured see the Oracle Documentation available at this link.

The example below shows the option to add an OCI Block Volume as an OLVM "Storage Domain"; the BV IP address is recognized automatically by Oracle Linux Virtualization Manager.

block-volume-01.png

KVM Virtual Machine creation

There are different ways to create one virtual-machine on OLVM / OL-KVM; the same can be created starting from an OVA or directly installed from an Operating-System ISO.

For further details on how-to import or create a Virtual Machine you can follow the steps available on Oracle Documentation at:

·       Creating a Virtual Machine

·       Creating a Virtual Machine from a Template

 

Network Management for Oracle Linux KVM Virtual Machines

Note: following steps can be applied to each required Virtual Machine Virtual NIC creation

To proceed to correctly configure networking for your Oracle Linux KVM Virtual Machines, proceed with the following steps:

  • Create the Virtual NIC on OCI web interface respecting following parameters
    • Name: <use the VM-name so it will be easier to recognize which vNIC is used by which Virtual Machine>
    • Subnet: kvm-vm-01
    • Physical NIC: NIC 1 (a must)
    • Skip Source/Destination Check: enabled
    • Private IP address: <your choice>
    • Assign Public IP address: <optional>
    • Hostname: <hostname that will be used within the Virtual Machine>
  • Values example:
    • Name: vm05-vnic01
    • Subnet: kvm-vm-01
    • Physical NIC: NIC 1
    • Skip Source/Destination Check: enabled
    • Private IP address: 10.0.2.201
    • Assign Public IP address: Yes
    • Hostname: vm05

OCI vNIC information required in the following steps are:

  • OCI vNIC Private IP address
  • OCI vNIC Public IP address (if defined)
  • OCI vNIC Mac Address
  • OCI vNIC VLAN Tag
  • Get the same vNIC defined on OLVM / OL-KVM host.

Note: With OLVM running on OCI, each OLVM Network will correspond to an OCI vNIC that, used as passthrough device, will be dedicated to a single VM.

Standard bridging networking is not possible on OCI.

·       Open the OLVM web-interface, "Network" => "Networks" and click on "New" button

net-creation.png

On the Network creation window, supply following details:

  • Name: <use same name used on OCI, so it will be easier to identify the correct association>
  • Enable VLAN Tagging checkbox and insert the proper VLAN Tag you've on OCI vNIC
  • Leave all the other parameters on their default values

net-creation-02.png

·       Open the OLVM web-interface, "Network" => "vNIC Profiles", select the profile created for your "Network" (same name) and click on "Edit" button.

·       On the "Edit" Windows enable the "Passthrough" option as in the following example:

net-creation-03.png

·       Associate the OLVM Network with proper OCI NIC Virtual Function.

 

The target, here, is to associate an "OLVM Logical Network" to an "OCI Virtual Function".

DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

 

·       Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host.

·       Click on "Network Interfaces" tab and click on "Show Virtual Functions" button

show-vfunctions-01.png

Above you can see the list of "OCI Virtual Functions" available that can be used to associate the "OLVM Logical Network" created.

·       Click on the "Setup Host Networks" button to process the connection between the "OLVM Logical Network" and "OCI Virtual Function"; on the "Setup Host Networks" click on "Show Virtual Functions" checkbox

net-05.png

On the Picture above, you can see:

·       Interfaces (left): list of physical NICs (see SR-IOV logo) and Virtual Functions (see vFunc logo)

·       Assigned Logical Networks (center): Logical Networks created on OLVM associated to Physical NIC(s) or Virtual Functions

·       Unassigned Logical Networks (right): Logical Networks created and actually not associated to any NIC/Virtual Function

 

·       Drag your "Logical Network" to one of the empty and available "Virtual Functions"

passthrough-conf-02.png

·       Configure Virtual Machine vNIC with proper HW-ADDR, as supplied by OCI web interface

·       Open the OLVM web-interface, "Compute" => "Virtual Machines" and click on your VM name to open its details

vm-conf-01.png

·       On the "Virtual Machine Configuration" section, click on "Network Interfaces" tab and edit "vNIC Settings"

vm-conf-02.png

·       On the "Edit Network Interface" window, enable the "Custom MAC address" checkbox and insert the "HW Address" supplied by OCI for this "Virtual Function"

vm-conf-03.png

 

Start your OLVM/OL-KVM Virtual Machine

·      Open the OLVM web-interface, "Compute" => "Virtual Machines" and select the line that identifies your Virtual Machine and start the same.

vm-start-01.png

Useful options that you could apply to your environment:

  • for the Virtual Machine configuration use the FQDN supplied by OCI
  • on the Virtual Machine field "Comment" add the (optional) OCI Public IP address as a reference

Join the discussion

Comments ( 4 )
  • Frank LADEH-AHLIDZA Wednesday, November 4, 2020
    Hi Simon,

    Thanks for this document.
    I have a question about the following configuration.
    You said :"Oracle Linux Virtualization Manager, built on OL7 image (latest available), has following requirements:

    Built and created from latest OL7.7 (or higher but lower than OL8) image

    ·vNIC(1): associated to "OLVM" Public subnet => Public IP address enabled

    ·vNIC(2): associated to "OCI" Private subnet => No Public IP address"



    My question is : On witch physical NIC should be the Vnic(2) created.

    Thanks
  • Simon Wednesday, November 4, 2020
    Hi Frank,

    you can decide, there's no restriction on where to create vNIC(2).
    Thanks

    Simon
  • Frank LADEH-AHLIDZA Monday, November 9, 2020
    Hi

    I think there is an error in one of the command in OLVM install process
    I try "yum install ovirt-release -y" which is in this document with no succes.

    The command which finally work for me is "yum install oracle-ovirt-release-el7 -y"

    Thanks
  • Simon Monday, November 9, 2020
    Hi Frank,

    thanks for your message.
    I got it fixed.

    Simon
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.