Overview

This blog post is the fourth in a series on OCI break-glass (see Parts One, Two, and Three). In this post, we do an overview of the ServiceNow side of the break-glass design, then get into the ServiceNow implementation nitty-gritty.

The ServiceNow® Change Management Application

The ServiceNow Platform has many features and capabilities. In this post, we'll only focus on the essential knowledge needed to implement our break-glass design, starting with an overview of the key components of the ServiceNow "Change Management" application, which implements the ITIL-Level 1 Emergency Change process.

In simplified terms, the Change Management application consists of the following :

1. Table: change_request contains the data associated with the change requests and current process state.

2. Forms & Views: UI elements that enable users to capture data necessary for a Change Request and facilitate the transition of the Change Request through the process stages

3. Change Models: Defines the states, and state transition conditions of the ITIL Change processes, as well as predefined data elements of the change requests. ServiceNow provides a default Emergency Change model that defines the ITIL Level-1 Emergency Change process stages and transitions.

4. FlowDesigner Flows: Defines the control logic of a business process in ServiceNow. Several predefined Flows are provided and are triggered during the various relevant stages of the Emergency Change process. The flows of the Emergency Change process are: Change - Emergency - Implement, Change - Emergency - Authorize, Change - Emergency - Review

5. Sub Flows, Actions: Implements granular reusable logic callable from the Flows.

The broad strokes of how these components work together in the ServiceNow Emergency Change process is that as authorized, ServiceNow users capture Emergency Change request data and move a change request through the process stages using the UI elements. The underlying change_request table is updated to reflect, among other things, the current state of the process (New, Authorized, Deployed, etc.). Flows are triggered when certain conditions are met in the change_request table (e.g., the process enters the Authorized state and is not "on-hold" ). These Flows automate the Emergency Change process by invoking Sub-Flows and Actions (e.g. making an outbound REST call).

Modifications to the Change Management Application

Now that we have a firm understanding of the basics, we can understand how to extend the ServiceNow Change Management application:

1. We'll make a copy of the default Emergency Change Model, adding "Record presets" for the level of "urgency" and the "priority" of the Emergency Changes created from the new model. We'll also modify the rules that govern transitions between process states to transition to the Implementation state after the change has been Approved and Scheduled ( this is not the default behavior of the Emergency Change Model in the ServiceNow Tokyo release).

2. We'll activate the ServiceNow "Integration Hub".

3. We'll configure Integration Hub "Outbound Connections" to connect and securely authenticate with the BreakGlass service. To make sure that requests to the BreakGlass service originate from approved ServiceNow instances, we'll implement the OAuth Authorization Code grant as well as mTLS.

4. Two new "Flow Designer" Actions will be created to invoke the BreakGlass service endpoints using the configured Integration Hub outbound REST connections. These Actions will specify the exact API endpoints on the BreakGlass service, with the necessary query parameters and payloads, to break and repair glass.

5. Finally, we'll modify two existing Emergency Change flows to invoke the new Actions, at the appropriate process stages. We'll also handle BreakGlass service responses and errors.

Let's jump into the nitty-gritty of how to implement this in the ServiceNow Tokyo release.

1. Create the Break-Glass Change Model

1.1. Navigate to the Change Management Application's Change > Change Models  module

1.2. Select the existing Emergency Change Model in the list

1.3. Make a copy of the existing Emergency Change Model

1.4. Rename the model and configure new "Record presets" for the Urgency and Priority fields ,as illustrated, then save the new model.

1.5. Modify the Authorize and Scheduled state configurations as shown below

2. Activate the ServiceNow Integration Hub Starter Pack

The steps to activate the Integration Hub Packs differ for developer instances versus production instances. This ServiceNow documentation link explains the steps to activate Integration Hub Packs. Follow the steps to activate the "ServiceNow IntegrationHub Installer". This will activate the "Starter Pack", which provides the outbound REST features we'll use in our implementation.

3. Configure Integration Hub Outbound Connections

3.1. Configure Mutual TLS (mTLS)

3.1.1. Setup the client certificate

To use mTLS authentication in ServiceNow, we will need to create a keystore file in a format that ServiceNow supports. This keystore must contain an RSA public/private key pair, a matching digital certificate for the client , and a keystore password.

Configure a new Certificate definition in ServiceNow by navigating to the System Definitions > Certificates module

Click the "New" button to start configuring a new certificate, attach the keystore and save the new certificate

3.1.2. Define a Protocol Profile

Administrators can enable mutual TLS authentication via "Protocol Profiles" that are used in ServiceNow to link certificates to a specific protocol, like HTTPS.

Navigate to the System Security > Protocol Profiles module

Select the keystore that we configured in ServiceNow in the previous step and save the Protocol Profile .We chose mtls as the protocol name in this example, but it can be any arbitrary protocol name.

3.2. Configure OAuth

As mentioned, the BreakGlass service will use OAuth 2.0 "Access Tokens" to authorize access to its APIs. In these instructions, we'll use an OCI IAM domain as our "Authorization Server". In OAuth terms, ServiceNow is the "Client" application, and the BreakGlass Service the "Resource Server". To configure ServiceNow to use OAuth, we must first create an "OAuth Provider" and "OAuth Entity Profile". Afterward, we can configure a "ServiceNow Credential" and "Credential Alias". Finally, we should configure an outbound ServiceNow "Connection" with a reference to the Credential Alias for authorization.

3.2.1. Configure two OCI IAM Domain "Confidential" Applications

Before configuring ServiceNow OAuth, we have to create two "Confidential" OAuth applications in an OCI IAM domain. One application should be configured as a "Resource Server" to define the OAuth configuration associated with our BreakGlass Service, including OAuth "scopes" and 'audience". The other should be configured as a "Client" representing the ServiceNow OAuth client..  

3.2.2. Configure a ServiceNow OAuth Provider and OAuth Entity Profile

Set up ServiceNow to retrieve OAuth "Access" and "Refresh" tokens using the OAuth Authorization Code Grant. 

3.2.2.1.  In ServiceNow navigate to the System OAuth > Application Registry module.

3.2.2.2.  Click the "New" Button and select the "Connect to a third party OAuth Provider" option

3.2.2.3. Complete the OAuth Provider configuration:

A default "OAuth Entity Profile" will be added in the tab at the bottom of the form. 

3.2.3. Configure a ServiceNow Credential 

Now that the OAuth Provider configuration is complete, we need to create a ServiceNow "Credential" that references the profile.

3.2.3.1. Navigate to the IntegrationHub > Credentials module

3.2.3.2. Create a new Credential of type "OAuth 2.0 Credentials"

Selecting "Get OAuth Token" starts the OAuth Authorization Code flow. This flow requires an OCI Domain user to authenticate to the OCI IAM Domain that has the confidential client app (that we created earlier ) assigned to the user. ServiceNow displays a status message indicating it has retrieved an Access and Refresh token from the OCI IAM domain Authorization Server after completing the Authorization code flow.

3.3. Configure a ServiceNow outbound connection

3.3.1. Navigate to IntegrationHub > Connection & Credential Aliases then create a New Connection & Credential Alias

3.3.2.  After you submit the form, navigate back into the newly created Connection Alias

3.3.3. Create a new connection using our previously configured Credential and mtls protocol

4. Create two Flow Designer Actions

One Action will invoke the BreakGlass service to break-glass, and the second will repair-glass to revoke the access provisioned by break-glass.

4.1 Create the invoke_BreakGlass Action

4.1.1.  Navigate to the ServiceNow Process Automation > Flow Designer

4.1.2. Create a new "Action" for the break-glass call with the following inputs: cr number and principal,  which will receive the change request number and assignment group of the change request from the calling flow.

4.1.3. Add a REST step to the Action. Set it to call the BreakGlass API endpoint. In our implementation, the Action POSTs a JSON payload containing the cr number and principal that was passed to the Action upon invocation.

4.2 Create the invoke_RepairGlass Action

The configuration of the second Action to repair-glass is :

Test the Actions to verify that they can invoke your BreakGlass Service endpoints to break-and-repair-glass, and then publish our two Actions to make them active.

5. Create Two new Flows

5.1 Modify existing Change-Emergency-Implement flow

Recall from the second post, when we introduced the ServiceNow Emergency Change Model, we decided to provision the break-glass user and credentials at the transition from the Schedule to Implement state. The Change-Emergency-Implement flow is the ServiceNow provided flow that is configured to execute when the Emergency Change process transitions into the Implement state. We'll therefore copy, then modify this flow to call the invoke_breakglass action that we created previously, passing the change request number and assignment group as inputs.

5.1.1. To make a copy of the flow, open the flow in the ServiceNow Flow Designer, then select the top right navigation menu and choose "Copy flow" from the dropdown list of options

5.1.2. The remaining changes will add steps to invoke the invoke_breakGlass action and steps to send notification emails to the "assignment group" members that direct them to a link on the SecretService (introduced in the second post) where they can retrieve the provisioned break-glass credential. The final flow looks like this :

5.1.3. Configure the invoke_BreakGlass Action

5.1.4. Send an email notification (using the ServiceNow 'Send Email Action") to the change request assignment group's email address with a link to retrieve the provisioned break-glass credential from the SecretService.

5.2. Modify existing Change - Emergency - Review Flow

In previous posts, we decided to put the "repair-glass" steps at the point where the ITIL level 1 – Emergency Change process transitions into the Review state. The ServiceNow Flow that triggers when this transition to Review happens is Change-Emergency-Review. In this section, we'll make a copy of this Flow and then proceed to make some modifications to invoke the invoke_RepairGlass Action we defined above.

5.2.1. Open the Change-Emergency-Review Flow in the ServiceNow Flow Designer, then select the top right navigation menu and choose "Copy flow" from the dropdown list of options. Give the copy a new name "Change - Emergency - Break - Glass - Review". Change the trigger for the new Flow to fire when the "Model" is Break-Glass.

5.2.2. Add, and configure, a new invoke_RepairGlass Action to the Flow :

5.2.3. If the invoke_repairGlass Action fails, create a new Change Post Implementation task to manually remove the provisioned break-glass credential. We'll talk about how to do that in the next post in this series.

Conclusion

The reader should have enough information to create a proof of concept at this point. The implementation details of the BreakGlass and SecretService aren't essential since any implementation of these REST services that meet our stated requirements will do.

In the next post in the series, we will address some of the unresolved issues from the second and third posts. We'll start by discussing how to recover if the BreakGlass service experiences an emergency...how do we  break-glass BreakGlass without ending up with an infinite regress?