X

Using Work Requests to Monitor the Status of Compute Operations

Andy Corran
Principal Product Manager

Today, we are introducing work requests for Oracle Cloud Infrastructure Compute resources. Work requests give you visibility into the status of common compute operations: creating an instance, creating an image, importing and exporting an image, and any create, read, update, or delete (CRUD) operations on an instance pool. This new visibility gives you operation progress feedback and detailed activity logs for each control plane event.

Previously, you had no indication if an initiated compute operation was still running or failed. Now you can see the progress of each operation, whether it results in a failed state, which step it failed on, and the reason for the failure. This post explains how to get this information by retrieving work request activity logs.

How Work Requests Work

Compute creates work requests for long-running operations that require sequential operations or have multiple affected resources that spawn asynchronous requests within a workflow. The output from the requests are logged, categorized, and consolidated into either informational or error messages.

Work requests answer three key questions:

  • Which operation was run?
  • When did it start and finish?
  • What is its current state, and if it failed, how did it fail?

Oracle Cloud Infrastructure Work Requests Process

For each operation, you can access the associated work requests and the insights for each activity collected through the Console, the CLI, and the API.

Access Work Requests in the Console

In the Console, you access work requests for a supported compute resource by navigating to the resource's details and selecting Work Requests in the Resources section.

Oracle Cloud Infrastructure Work Requests

Work requests return a set of defined status conditions, which provide a real-time indication of the state and completion progress of any running work request activity. The log retention period for this initial release is 12 hours. You can download the work request logs for a given compartment though the API or CLI and warehouse accordingly. For future releases, we are planning additional capabilities, such as retaining selected activity logs for longer periods and integrating with export/stream with Oracle Cloud Infrastructure Cloud Events for log event ingestion.

Example: Viewing the State of an Instance Pool Work Request

When you create an instance pool, a workflow based on the pool parameters that you set is initiated. This section shows an example of creating an instance pool and getting insights into why its creation failed.

When you initiate the creation of an instance pool in the Console, you are directed to the work requests view, as shown in the following image, where real-time status and completion progress are displayed.

Oracle Cloud Infrastructure Create Instance Pool Work Request

All activities are logged by the work request workflow. When you view the work request details, you can see log messages, error messages, and associated resources. Log and error messages for each activity provide a status description and timestamp. In the following image, the log messages show that no instances were launched in the pool, and there is one error message.

Oracle Cloud Infrastructure Create Pool Work Request Failure

Work request error messages list failure log details. The following image shows that an instance launch failed because the service limit was reached in an availability domain for a VM.Standard2.1 shape. Recommended actions are also provided so that you can resolve the failure yourself, if possible.

Oracle Cloud Infrastructure Create Instance Pool Errors

Access Work Requests Through the CLI

You can use the Oracle Cloud Infrastructure CLI (quickstart install) to retrieve compute work request log activities for compartments that you have access to. The log output from the CLI is in a JSON format. You can simplify your command line experience with Microsoft PowerShell, which lets you to consume the work request activity logs as objects. You can filter, sort, iterate, download, and visually present in a readable format.

The following example shows how to retrieve the work request log by using the CLI and PowerShell together. The example scenario is the same as for the Console: creating an instance pool.

  1. From the PowerShell command line, get and set the compartment OCID variable. This is the compartment in which the instance pool is being created.

    Get list of compartments

    PS C:\>./oci iam compartment list | ConvertFrom-Json | Select-Object -ExpandProperty data | fl id, name

    Set Variable for compartment

    PS C:\> $Cmpt = "COPY COMPARTMENT OCID HERE”

  2. Get all work requests created for compute resources in the compartment.

    PS C:\> $WR = ./oci work-requests work-request list --compartment-id $Cmpt | ConvertFrom-Json

  3. Parse through all the work requests and store the selected work request ID in the $WR_ID variable:

    • Sort and order by date and time: time-accepted

    • Filter on operation-type: LaunchInstanceInPool to get only work requests for creating an instance pool.

    • Retrieve the last work request that ran and retrieve its ID.

    PS C:\> $WR_ID = $WR.data | sort-object -Property time-accepted | Where-object operation-type -eq "LaunchInstancesInPool" | select-object -last 1 -ExpandProperty id

  4. Get all the log entries for this work request ID and store them in the $WR_Log variable. Here you can use PowerShell to create a formatted list output of the work request log entries as objects.

    PS C:\> $WR_Log = ./oci work-requests work-request-log-entry list --work-request-id $WR_ID --all | ConvertFrom-Json

    PS C:\> $WR_Log.data | fl

    message : Launched 0 out of 1 requested instances.
    timestamp : 2019-05-31T22:00:11.453000+00:00

    message : Validating instances: 0 launched, 0 provisioning, 0 failed
    timestamp : 2019-05-31T22:00:01.121000+00:00

    message : Launching 1 instances for instance pool ocid1.instancepool.oc1.iad.aaaaaaaaldvx3ncwcli3pf2li4nnze3iybjniubatfy744pi35jycyhsutvq
    timestamp : 2019-05-31T21:59:33.896000+00:00

  5. Get all the error log entries for this work request.

    PS C:\> $WR_ErrLog = ./oci work-requests work-request-error list --work-request-id $WR_ID --all | ConvertFrom-Json

    PS C:\> $WR_ErrLog.data | fl

    code : 1600

    message : Instance launch failed. You have reached your service limit of 5 in this Availability Domain for VM.Standard2.1. Please try launching the instance in a different Availability Domain or Region, or try using a different shape. If you have reached all Service limits, please contact Oracle support to request a limit increase.

    timestamp : 2019-05-31T21:59:48.068000+00:00

  6. Put it all together in one output in a formatted list to view all the logs and errors for this single work request.

    PS C:\> Write-Output  "`nLOG:`n====" $WR_Log.data "`nERROR LOG:`n==========`n" $WR_ErrLog.data | fl

     

    LOG:

    ====

    message   : Launched 0 out of 1 requested instances.

    timestamp : 2019-05-31T22:00:11.453000+00:00

     

    message   : Validating instances: 0 launched, 0 provisioning, 0 failed

    timestamp : 2019-05-31T22:00:01.121000+00:00

     

    message   : Launching 1 instances for instance pool ocid1.instancepool.oc1.iad.aaaaaaaaldvx3ncwcli3pf2li4nnze3iybjniubatfy744pi35jycyhsutvq

    timestamp : 2019-05-31T21:59:33.896000+00:00

     

    ERROR LOG:

    ==========

    code : 1600

    message   : Instance launch failed. You have reached your service limit of 5 in this Availability Domain for VM.Standard2.1. Please try launching the instance in a different Availability Domain or Region, or try using a different shape. If you have reached all Service limits, please contact Oracle support to request a limit increase.

     timestamp : 2019-05-31T21:59:48.068000+00:00

With just a few commands, you can view the state of a work request and the result of each activity. PowerShell for Windows users provides the same flexibility to manipulate the execution of CLI specific commands, much in the same way on Linux.

Access Work Requests Through the API

You can monitor the state of work request log activities by using the following REST API operations:

Next Steps

This new feature gives you increased visibility into the status of compute activities. Work requests let you monitor activity progress with the benefit of logging details to help you troubleshoot problems if an activity fails to complete.

To try it out and learn more, see the following documentation:

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha