Oracle Container Engine for Kubernetes (OKE) now supports customizing Kubernetes worker nodes using startup scripts. This functionality provides users greater control over configuring the worker node hosts that run Kubernetes workloads.

Background

Cloud-init is the industry standard method for initialization and configuring cloud instances. It has support across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare metal installations. For more information about cloud-init, see the cloud-init documentation.

OKE installs a default cloud-init startup script on each Oracle Cloud Infrastructure (OCI) Compute instance, which configures the instance as a Kubernetes worker node. When the instance boots up the cloud-init script runs. The default startup script contains the following logic:

#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh

The default host configurations set by OKE don’t necessarily match every use case, which can lead to a need for you to make your own customizations to worker nodes. Previously, you could customize worker nodes either one at a time or at scale by connecting to worker nodes by SSH, making changes, and then saving those changes as a custom image to be used as the template for future worker nodes.

Use cases

You can customize the default startup script by adding your own logic to the script, either before or after the OKE logic runs. This capability gives you the flexibility to perform certain actions that need to be run before the running of the OKE script. For example, you can configure an internal yum repository or set up a corporate proxy.

Customizing the default startup script enables many use cases, including the following examples:

  • Use kubelet-extra-args to configure kubelet taints to control which nodes pods are scheduled onto.

  • Configure a security-enhanced Linux (SELinux) policy on all worker node hosts for security and compliance purposes.

  • Unassign an instance’s ephemeral public IP on start-up, and reassign the instance a reserved public IP instead.

  • Configure a corporate proxy.

  • Configure custom yum proxies.

  • Install mandated antivirus software and other security tools.

You can customize the default startup script when creating a cluster, creating node pools, and modifying existing node pools. The customized startup script runs when an instance hosting a worker node is booted up. After customizing the default startup script, best practice suggests running the Node Doctor script to confirm that worker nodes on newly started instances are working as expected. For more information, refer to Node Troubleshooting Using the Node Doctor Script.

Using a custom cloud-init script to set kubelet-extra-args

You can use a custom cloud-init script to configure several extra options on the kubelet (the primary node agent) on worker nodes. These extra options are sometimes referred to as kubelet-extra-args. One kubelet- extra-args option is configuring debug level log verbosity.

To configure debug level log verbosity, use the following cloud-init script:

>
#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh --kubelet-extra-args "--v=4"

To confirm the setting of debug level log verbosity, connect to a worker node and use the sudo systemctl status -l kubelet command. When the cloud-init script has been run on worker nodes, the sudo systemctl status -l kubelet command returns the verbosity level as 4. The kubelet logs also contain more detail

Using a custom cloud-init script to configure SELinux

SELinux is a security enhancement to Linux that enables administrators to constrain which users and applications can access which resources based on rules in a policy. SELinux also adds finer granularity to access controls. You can configure SELinux on worker nodes using a custom cloud-init script.

SELinux can be in one of two states, either enabled or disabled. When enabled, SELinux can run in one of two modes, either enforcing or permissive. By default, SELinux is enabled on OKE worker nodes and set to run in permissive mode. When running in permissive mode, SELinux doesn’t enforce access rules and only performs logging.

To enforce access rules, set SELinux to run in enforcing mode. When running in enforcing mode, SELinux blocks actions that are contrary to the policy and logs a corresponding event in the audit log.

To set SELinux to run in enforcing mode, use the following cloud-init script:

>
#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh
setenforce 1
sed -i ’s/^SELINUX=.*/SELINUX=enforcing/’ /etc/selinux/config

 

To confirm the status and mode of SELinux running on a worker node, connect to the worker node and use the getenforce command. When the cloud-init script has run on worker nodes, the getenforce command returns Enforcing.

Conclusion

Customizing worker nodes using startup scripts enables advanced users to tailor-make worker nodes that match their use case. You’re no longer stuck deciding between whether to use the default node configuration or accept the extra management overhead of manually managing custom images. Now you can simply add a custom script before or after the OKE script runs to modify your worker nodes as you see fit.

To learn more or get hands-on, use the following resources: