Normally, diagnosing Compute instances that fail to boot or become non-responsive is difficult. A common first step to troubleshooting is to connect to the instance through SSH or RDP. However, if the instance’s operating system failed to start, the services required for SSH or RDP access isn’t running.

We’ve launched several Oracle Cloud Infrastructure (OCI) Console features that make diagnosing these issues easier by extending support for the serial console. If you’re familiar with on-premises servers or personal computers, the serial console is the text output you see when you connected a monitor to the serial port on the device. Instances in OCI can output information to a virtualized serial console. The web Console now supports viewing the serial console history for a given instance and an easy, one-click interactive serial console connection using Cloud Shell. These features can be useful to anyone debugging, operating, or maintaining instances in OCI.

In this blog post, we use both these newly added serial console features to demonstrate how to debug an issue with an Oracle Linux 8 instance. If you have a custom Linux image, you might need to enable serial console access. If you have a Windows image, you can instead enable the Special Administration Console (SAC), part of Windows’ Emergency Management Services. If any given step doesn’t work, refer to the documentation and ensure that you completed the prerequisite steps.

Viewing an Oracle Linux 8 instance’s console history using the OCI Console

From the Instances list page, locate the instance that you want to diagnose. In the Resources list in the lower left, select Console history and then click the View current history button. This button captures a snapshot of the most recent output (up to one megabyte) of serial console output and displays it in a scrollable panel. Errors can show up here during boot and are presented in red text. You can use your browser’s search functionality or download the file for searching in the IDE or word processor of your choice.

A screenshot of the example Console history.

However, even if we identify the problem from the serial console output, we can’t resolve the issue using SSH or RDP, since we can’t access the host.

Recovering an Oracle Linux 8 instance using an interactive serial console through the OCI Console

Instead of accessing the instance by SSH or RDP to correct the issue, we can use an interactive serial console session. From the Instances list page, locate the instance you diagnosed. In the Resources list in the lower left, select Console connection and then click Launch Cloud Shell connection. This selection opens Cloud Shell, an in-browser shell experience, and then runs a script that sets up an interactive serial console session. Starting Cloud Shell can take several minutes.

A screenshot of the Console Connection screen with Cloud Shell open at the bottom.

Now that we have an interactive serial console session, we can reboot into maintenance mode. If the following steps don’t work, refer to the full documentation.

  1. Reboot the instance from the Console using the Reboot button.

  2. Click the Cloud Shell terminal and repeatedly press ESC or F5 while the host reboots.

  3. In the boot menu, highlight the top item and press E to edit it.

  4. In edit mode, use the down arrow key to scroll down through the entries until you reach the line that starts with Linux. If the arrow keys don’t work, you can use the hotkeys.

  5. At the end of that line, add the following: ’init=/bin/bash’

  6. Reboot the instance from the terminal window by entering the keyboard shortcut, CTRL+X.

  7. After reboot, you’re in an interactive bash shell. You can solve the issue you identified earlier using any of the available bash built-ins and binaries on your path. For example, you might resolve an issue by installing a missing dependency with package management software (‘yum’ or ‘apt-get’), changing a file’s permissions with ‘chmod,’ or by freeing up space with ‘rm.’ If you need to make changes to the filesystem, such as installing software, changing permissions, or deleting files, you first need to remount the filesystem as read/write instead of read-only with the command, ‘mount -o remount,rw /.’

  8. After resolving your problem, reboot the instance using the Reboot button in the Console.

Conclusion

While these two new tools—the read-only serial console history and the live serial console session through Cloud Shell—can make it significantly easier to debug and recover instances, they’re only part of Oracle Cloud Infrastructure’s debugging toolkit. You can find more strategies in the Troubleshooting Compute Instances docs page. The Best Practices for Compute Instances and Compute Known Issues docs pages can also help you troubleshoot common issues.