The IT Employee Support team at Oracle handles about 50,000 service tickets monthly, with a peak of about 5,000 tickets daily. Employees report issues in Slack, and the integrated Atlassian Assist Slack bot, based on a rule-based system using keywords, provides initial answers. Because the current bot underperformed in multiple aspects, including its inability to accurately parse the prompts and generate relevant recommendations of knowledge articles, more than 90% of bot engagements resulted in tickets that need to be handled by 140 IT support engineers worldwide. Because this ticket deflection rate is far from ideal, we started a redesign of the bot using Generative AI and aimed to improve efficiency of our IT support engineers. 

By studying the historical bot interactions and tickets, we determined that the improvements should focus on the following inefficiencies of the previous chatbot system:

  • Limited grasp of context: It struggled to parse and maintain context during a conversation, leading to inaccurate responses, especially when dealing with complex queries.
  • Handling ambiguity and vagueness: Ambiguous or vague user queries were challenging for the chatbot to interpret accurately. In such cases, the chatbot gave generic responses, which frustrated users, which led to creating support tickets that prolonged issue resolution.
  • Extended response times: Higher ticket volumes resulted in longer response times. Employees experienced delays getting the help they needed, contributing further to frustration and dissatisfaction. These factors negatively affected employee experience and productivity as they perceived a decline in the quality of service, impacting their satisfaction and engagement.

Overview

To take advantage of generative AI capabilities, we built the AI Service Desk, an intelligent Slack bot that provides accurate, step-by-step solutions to user inquiries, handles many tickets without compromising quality, automatically deflects support issues, and continuously builds from user interactions to improve relevance. 

The AI Service Desk uses an internal knowledge base to answer questions through a retrieval-augmented generation (RAG) model architecture. This architecture is based on the search component, which helps to narrow the information space, making the generative part more focused and effective. This service has been built using the following Oracle services and features:

  • Oracle Autonomous Database, serving as the IT data lake source that includes data from multiple knowledge sources, such as help knowledge articles, Confluence pages, and intranet web pages
  • Oracle Digital Assistant for chat interactions
  • Oracle Mobile Hub for mobile app integration
  • Oracle Autonomous Database for storing metrics used for analytics
  • RAG solution based on Oracle Cloud Infrastructure (OCI) Generative AI, providing the large language model (LLM) service
  • OCI OpenSearch for indexing knowledge sources and searching efficiently
  • Various machine learning (ML) models and accuracy measurements using OCI Data Science

AI Service Desk was released early 2024, and we continuously evaluate the effectiveness across key metrics namely, percent of queries resolved by the new AI solution, percent increase in ticket deflection rate, percent reduction in ticket volume, ticket resolution time, employees’ satisfaction, and support engineers’ productivity.

“With Generative AI powered self-service features in the AI Service Desk, we noticed ticket deflection rates of 25–30%,” said Sudhir Dureja, senior director of Enterprise Engineering. This rate is equivalent to 3100–4000 tickets per week.

This improved deflection rate has allowed support engineers to invest more time in enhancing the knowledge base to improve the effectiveness of the chatbot and for providing more specialized Tier 2 support services to employees.

Implementation details

Most work on the AI Service Desk builds on top of the years of our investment in building a modern search platform for the Oracle intranet, where we ingest data from 13 different services and handle over 500,000 queries per month with an exceptionally high 75% click-through rates.

The AI Service Desk has following key modules:

  • Slack bot based on Oracle Digital Assistant
  • Ticketing system used by support engineers
  • OCI Generative AI based on a RAG solution
  • Common analytics system using APEX
Figure 1: AI Service Desk high-level flow diagram
Figure 1: AI Service Desk high-level flow diagram

Slack bot based on Oracle Digital Assistant

Employees who need help from the support team ask questions and seek help through an internal Slack channel call #it-support. A Slack custom app, called AI Service Desk, has been configured to monitor all messages being posted into this channel and can respond autonomously to those messages. The Slack app is configured to use a webhook URL from the Oracle Digital Assistant Slack Channel configuration.

When a user inputs a query with the bot, they go through the following typical flow:

  1. The Slack bot greets the employee. 

    The Slack bot greets the employee.

     
  2. The bot processes the user’s question, automatically generating a prompt and passes it to the RAG solution to get a grounded response.
  3. It provides the employee detailed step-by-step instructions generated by the OCI Generative AI service using the knowledge base. 

    t provides the employee detailed step-by-step instructions

     
  4. Two action buttons are also provided.
    • View article: This citation of knowledge source is based on the answer that was generated. It allows users to review the article if they need more information
    • Contextual dynamic action: For example, Reset SSO password. The action button configured with the most relevant knowledge article is presented to help the user act directly.
  5. At this point in the flow, employees follow the instructions and take action. The bot then prompts to confirm whether the issue has been resolved. If the employee confirms, “This solved my issue,” the Slack thread is closed, and no more messages are permitted on that thread.

    The bot then prompts to confirm whether the issue has been resolved

     
  6. If the employee selects, “I still need help,” a form is shown. Upon completion, the form leads to the creation of a support ticket (SR), which is assigned to a human support engineer. The assigned support engineer uses the integrated ticketing system (described in the next section) to engage the employee to resolve the service request. With native integration, the employee and support engineer can continue to engage in the same Slack thread to retain continuity and context until the ticket is resolved, removing the need for switching to separate systems to log and progress support tickets.

    Action buttons are provided to confirm if the issue is resolved or not

Ticketing system used by support engineers

Human support engineers use Atlassian Jira Service Management for tracking and serving the service requests. The Oracle Digital Assistant-based Slack bot is integrated with Jira through skill configuration, which enables the Oracle Digital Assistant agent to pass all conversations from the Slack channel to the support engineer using a Jira ticket. The support engineer interacts back and forth with the employee using the Jira console, while the user responds in the Slack thread (see Figure 2). This integration helps the employee and the support engineer to interact through the Slack bot and Jira, which maintains continuity and avoids friction of using each other’s applications.

Figure 2: Slack thread interactions between the support engineer and employee
Figure 2: Slack thread interactions between the support engineer and employee

 

OCI Generative AI-based RAG solution

The user-entered question and context are passed to the retriever system, which performs a semantic search on the large volume of indexed articles and retrieves the top ranked documents. Those top ranked documents and the user’s question-based prompt are passed to the Q&A model, which uses OCI Generative AI to produce the best-extracted answer. The best-extracted answer and the document ID are passed back to the bot, which responds to the employee. The following figure shows the high-level flow diagram of how RAG architecture is implemented.

Figure 3: High-level flow diagram for the RAG architecture
Figure 3: High-level flow diagram for the RAG architecture

 

Common analytics system

So far, we have seen how the Oracle Digital Assistant-based Slack bot, ticketing system, and Generative AI-based RAG solution work together to automate IT support. The efficacy of this AI Service Desk bot relies on continuous monitoring of its answers, ticket deflection rates, and the time to ticket resolution. The common analytics system is built to track all activities of this AI Service Desk bot, including the employee and the support engineer interactions and aggregate them in the Oracle Autonomous Database. A custom Oracle APEX application calculates metrics including number of threads, tickets created and deflected from the Oracle Autonomous Database data and visualizes the information in the dashboard as shown in Figure 4.

Figure 4: Analytics showing total ticket volumes and ticket deflections
Figure 4: Analytics showing total ticket volumes and ticket deflections

 

The Support Analysts team and the Data Science team at Oracle Support organization monitor this common analytics dashboard. The Support Analysts team use this dashboard to track the service inquiries and analyzes the top ticket categories that Generative AI can’t deflect. If the deflected tickets are caused by the lack of knowledge sources, then the life cycle of knowledge article updates are analyzed to determine the wanted knowledge article improvement to increase ticket deflection rate.

The Data Science team uses the same dashboard in parallel to analyze the user interactions to identify ways to optimize the RAG model.

Figure 5: Report showing the top-tier ticket data
Figure 5: Report showing the top-tier ticket data

 

The future of the AI Service Desk

We aim to improve ticket deflections to 50–60% over the next 6–12 months by continuing to invest in the RAG model and improving the knowledge base. Further development continues on the action framework to enable the AI Service Desk to orchestrate remediation based on user questions and known employee context information, resolving employee issues without the need for human support engineer intervention. The action framework is intended to improve the employee experience and resolve employee issues even faster.

For more information on the concepts in this post, see the following resources: