OCI Break-Glass , Part Four

March 13, 2023 | 16 minute read
Gordon Trevorrow
Principal Cloud Architect
Text Size 100%:


This blog post is the fourth in a series on OCI break-glass (see Parts One, Two, and Three). In this post, we do an overview of the ServiceNow side of the break-glass design, then get into the ServiceNow implementation nitty-gritty.

The ServiceNow® Change Management Application

The ServiceNow Platform has many features and capabilities. In this post, we'll only focus on the essential knowledge needed to implement our break-glass design, starting with an overview of the key components of the ServiceNow "Change Management" application, which implements the ITIL-Level 1 Emergency Change process.

In simplified terms, the Change Management application consists of the following :

1. Table: change_request contains the data associated with the change requests and current process state.

2. Forms & Views: UI elements that enable users to capture data necessary for a Change Request and facilitate the transition of the Change Request through the process stages

3. Change Models: Defines the states, and state transition conditions of the ITIL Change processes, as well as predefined data elements of the change requests. ServiceNow provides a default Emergency Change model that defines the ITIL Level-1 Emergency Change process stages and transitions.

4. FlowDesigner Flows: Defines the control logic of a business process in ServiceNow. Several predefined Flows are provided and are triggered during the various relevant stages of the Emergency Change process. The flows of the Emergency Change process are: Change - Emergency - Implement, Change - Emergency - Authorize, Change - Emergency - Review

5. Sub Flows, Actions: Implements granular reusable logic callable from the Flows.

The broad strokes of how these components work together in the ServiceNow Emergency Change process is that as authorized, ServiceNow users capture Emergency Change request data and move a change request through the process stages using the UI elements. The underlying change_request table is updated to reflect, among other things, the current state of the process (New, Authorized, Deployed, etc.). Flows are triggered when certain conditions are met in the change_request table (e.g., the process enters the Authorized state and is not "on-hold" ). These Flows automate the Emergency Change process by invoking Sub-Flows and Actions (e.g. making an outbound REST call).

Modifications to the Change Management Application

Now that we have a firm understanding of the basics, we can understand how to extend the ServiceNow Change Management application:

1. We'll make a copy of the default Emergency Change Model, adding "Record presets" for the level of "urgencyand the "priority" of the Emergency Changes created from the new model. We'll also modify the rules that govern transitions between process states to transition to the Implementation state after the change has been Approved and Scheduled ( this is not the default behavior of the Emergency Change Model in the ServiceNow Tokyo release).

2. We'll activate the ServiceNow "Integration Hub".

3. We'll configure Integration Hub "Outbound Connections" to connect and securely authenticate with the BreakGlass service. To make sure that requests to the BreakGlass service originate from approved ServiceNow instances, we'll implement the OAuth Authorization Code grant as well as mTLS.

We do not want to turn ServiceNow administrator credentials into a proxy for the OCI break-glass credential. An authorized ServiceNow administrator must not be able to bypass the process by duplicating the security configurations of the outbound ServiceNow connection in another REST client to invoke the BreakGlass service endpoints directly. AuthZ Code Grant together with mTLS allows for the Segregation Of Duties when configuring the security for the outbound connections. We'll talk more about this and some additional ServiceNow security concerns later

4. Two new "Flow Designer" Actions will be created to invoke the BreakGlass service endpoints using the configured Integration Hub outbound REST connections. These Actions will specify the exact API endpoints on the BreakGlass service, with the necessary query parameters and payloads, to break and repair glass.

5. Finally, we'll modify two existing Emergency Change flows to invoke the new Actions, at the appropriate process stages. We'll also handle BreakGlass service responses and errors.

Let's jump into the nitty-gritty of how to implement this in the ServiceNow Tokyo release.

1. Create the Break-Glass Change Model

1.1. Navigate to the Change Management Application's Change > Change Models  module

Navigate to the Change Models view

1.2. Select the existing Emergency Change Model in the list

Select the existing Emergency Change Model

1.3. Make a copy of the existing Emergency Change Model

Make a copy of the Emergency Change Model

1.4. Rename the model and configure new "Record presets" for the Urgency and Priority fields ,as illustrated, then save the new model.


Rename and Save

1.5. Modify the Authorize and Scheduled state configurations as shown below


Break-Glass Model State transition changes

2. Activate the ServiceNow Integration Hub Starter Pack

The steps to activate the Integration Hub Packs differ for developer instances versus production instances. This ServiceNow documentation link explains the steps to activate Integration Hub Packs. Follow the steps to activate the "ServiceNow IntegrationHub Installer". This will activate the "Starter Pack", which provides the outbound REST features we'll use in our implementation.

Activate Integration Hub Starter Pack

3. Configure Integration Hub Outbound Connections

3.1. Configure Mutual TLS (mTLS)

3.1.1. Setup the client certificate

To use mTLS authentication in ServiceNow, we will need to create a keystore file in a format that ServiceNow supports. This keystore must contain an RSA public/private key pair, a matching digital certificate for the client , and a keystore password.

The exact steps of generating a key pair, CSR, and client certificate are outside the scope of this article. We assume the reader knows how to create a supported keystore that contains a client certificate from a Certificate Authority that the BreakGlass Service trusts.

Configure a new Certificate definition in ServiceNow by navigating to the System Definitions > Certificates module

System Definition > Certificate

Click the "New" button to start configuring a new certificate, attach the keystore and save the new certificate

Configure Client Cert Keystore

3.1.2. Define a Protocol Profile

Administrators can enable mutual TLS authentication via "Protocol Profiles" that are used in ServiceNow to link certificates to a specific protocol, like HTTPS.

Navigate to the System Security > Protocol Profiles module

System Security> Protocol Profile

Select the keystore that we configured in ServiceNow in the previous step and save the Protocol Profile .We chose mtls as the protocol name in this example, but it can be any arbitrary protocol name.

Protocol Profile

3.2. Configure OAuth

As mentioned, the BreakGlass service will use OAuth 2.0 "Access Tokens" to authorize access to its APIs. In these instructions, we'll use an OCI IAM domain as our "Authorization Server". In OAuth terms, ServiceNow is the "Client" application, and the BreakGlass Service the "Resource Server". To configure ServiceNow to use OAuth, we must first create an "OAuth Provider" and "OAuth Entity Profile". Afterward, we can configure a "ServiceNow Credential" and "Credential Alias". Finally, we should configure an outbound ServiceNow "Connection" with a reference to the Credential Alias for authorization.

3.2.1. Configure two OCI IAM Domain "Confidential" Applications

Before configuring ServiceNow OAuth, we have to create two "Confidential" OAuth applications in an OCI IAM domain. One application should be configured as a "Resource Server" to define the OAuth configuration associated with our BreakGlass Service, including OAuth "scopes" and 'audience". The other should be configured as a "Client" representing the ServiceNow OAuth client..  

Resource Server Confidential App

Client Confidential App


3.2.2. Configure a ServiceNow OAuth Provider and OAuth Entity Profile

Set up ServiceNow to retrieve OAuth "Access" and "Refresh" tokens using the OAuth Authorization Code Grant.  In ServiceNow navigate to the System OAuth > Application Registry module.

Navigate to the Application Registry Module  Click the "New" Button and select the "Connect to a third party OAuth Provider" option

Select Connect to a third party OAuth Provider Complete the OAuth Provider configuration:

Configure ServiceNow OAuth Provider

A default "OAuth Entity Profile" will be added in the tab at the bottom of the form. 

Configure ServiceNow OAuth Provider


3.2.3. Configure a ServiceNow Credential 

Now that the OAuth Provider configuration is complete, we need to create a ServiceNow "Credential" that references the profile. Navigate to the IntegrationHub > Credentials module

IntegrationHub Credential Create a new Credential of type "OAuth 2.0 Credentials"

Select OAuth 2.0 Credentials

Configure OAuth 2.0 Credentials

Selecting "Get OAuth Token" starts the OAuth Authorization Code flow. This flow requires an OCI Domain user to authenticate to the OCI IAM Domain that has the confidential client app (that we created earlier ) assigned to the user. ServiceNow displays a status message indicating it has retrieved an Access and Refresh token from the OCI IAM domain Authorization Server after completing the Authorization code flow.

3.3. Configure a ServiceNow outbound connection

3.3.1. Navigate to IntegrationHub > Connection & Credential Aliases then create a New Connection & Credential Alias

Connection and Credential Alias

Connection & Credential Alias 2

3.3.2.  After you submit the form, navigate back into the newly created Connection Alias

Navigate back into newly created connection alias

3.3.3. Create a new connection using our previously configured Credential and mtls protocol

New connection 1Create connection 2


4. Create two Flow Designer Actions

One Action will invoke the BreakGlass service to break-glass, and the second will repair-glass to revoke the access provisioned by break-glass.

4.1 Create the invoke_BreakGlass Action

4.1.1.  Navigate to the ServiceNow Process Automation > Flow Designer

Flow Designer

4.1.2. Create a new "Action" for the break-glass call with the following inputs: cr number and principal,  which will receive the change request number and assignment group of the change request from the calling flow.

Break-Glass Action input

4.1.3. Add a REST step to the Action. Set it to call the BreakGlass API endpoint. In our implementation, the Action POSTs a JSON payload containing the cr number and principal that was passed to the Action upon invocation.

Add REST Action step

Select REST step

Config BreakGlass Action REST step


4.2 Create the invoke_RepairGlass Action

The configuration of the second Action to repair-glass is :

Repair-glass action inputRepair-glass REST step

Test the Actions to verify that they can invoke your BreakGlass Service endpoints to break-and-repair-glass, and then publish our two Actions to make them active.

5. Create Two new Flows

5.1 Modify existing Change-Emergency-Implement flow

Recall from the second post, when we introduced the ServiceNow Emergency Change Model, we decided to provision the break-glass user and credentials at the transition from the Schedule to Implement state. The Change-Emergency-Implement flow is the ServiceNow provided flow that is configured to execute when the Emergency Change process transitions into the Implement state. We'll therefore copy, then modify this flow to call the invoke_breakglass action that we created previously, passing the change request number and assignment group as inputs.

5.1.1. To make a copy of the flow, open the flow in the ServiceNow Flow Designer, then select the top right navigation menu and choose "Copy flow" from the dropdown list of options

Copy the flow 1

Copy the flow 2


Set Flow trigger to trigger on a Break-Glass change model

5.1.2. The remaining changes will add steps to invoke the invoke_breakGlass action and steps to send notification emails to the "assignment group" members that direct them to a link on the SecretService (introduced in the second post) where they can retrieve the provisioned break-glass credential. The final flow looks like this :



5.1.3. Configure the invoke_BreakGlass Action

configure invoke_BreakGlass inputs5.1.4. Send an email notification (using the ServiceNow 'Send Email Action") to the change request assignment group's email address with a link to retrieve the provisioned break-glass credential from the SecretService.

Config e-mail to Assignment group with link to retrieve JIT credential

5.2. Modify existing Change - Emergency - Review Flow

In previous posts, we decided to put the "repair-glass" steps at the point where the ITIL level 1 – Emergency Change process transitions into the Review state. The ServiceNow Flow that triggers when this transition to Review happens is Change-Emergency-Review. In this section, we'll make a copy of this Flow and then proceed to make some modifications to invoke the invoke_RepairGlass Action we defined above.

5.2.1. Open the Change-Emergency-Review Flow in the ServiceNow Flow Designer, then select the top right navigation menu and choose "Copy flow" from the dropdown list of options. Give the copy a new name "Change - Emergency - Break - Glass - Review". Change the trigger for the new Flow to fire when the "Model" is Break-Glass.

Trigger for modified review flow5.2.2. Add, and configure, a new invoke_RepairGlass Action to the Flow :

Review Flow overview

invoke repair-glass

5.2.3. If the invoke_repairGlass Action fails, create a new Change Post Implementation task to manually remove the provisioned break-glass credential. We'll talk about how to do that in the next post in this series.

Handle repair glass error


The reader should have enough information to create a proof of concept at this point. The implementation details of the BreakGlass and SecretService aren't essential since any implementation of these REST services that meet our stated requirements will do.

In the next post in the series, we will address some of the unresolved issues from the second and third posts. We'll start by discussing how to recover if the BreakGlass service experiences an emergency...how do we  break-glass BreakGlass without ending up with an infinite regress?

Gordon Trevorrow

Principal Cloud Architect

Previous Post

Difference between VB(VBCS) Vs VBS

Manoj Shetty | 16 min read

Next Post

Using OCI devops and terraform to synchronize an OCI bucket with a git repository

Christian Weeks | 10 min read