TechExchange: How-to allow customers to provide feedback on the usefulness of answers to frequently asked questions

article by Frank Nimphius, May 2020
updated, January 2022

Sometimes a question is what it is; a question. In this case, it makes little sense for a bot to start a long conversation with the user. Instead, the bot should give a direct answer to the question. Answer intents are a recent addition to Oracle Digital Assistant and use the same machine learning model to understand the user question as regular intents do. With answer intents, Oracle Digital Assistant provides a very reliable and successful implementation of the question-answer user case.

A common feature of web-based FAQ pages is that at the end of an answer the user has the opportunity to evaluate the quality of the answer and to provide feedback. Just recently, teams working with answer intents started requesting documentation for a similar implementation pattern for questions answered by Oracle Digital Assistant.

In this article I provide a channel independent sample implementation that behaves like answer intents in Oracle Digital Assistant but that allows extension to be added for users to provide feedback and for the answer response itself to optionally add channel specific properties.

Note: To avoid confusion between the answer intent feature in Oracle Digital assistant and the tip in this article, I refer to the question/answers solution this article provides as FAQ questions, FAQ answers or used FAQ.

The Sample In Action

The screen shots below show the sample skill that you can download at the end of this article exposed on the Oracle Web Channel. When a user starts a conversation (which could be through an initial message as shown in the image below, or through a message displayed in the messenger upon start-up) the sample displays a welcome message. The welcome message is triggered by a welcome intent, which means that it is part of a conversation that allows new users to onboard easily.

In this example, the user asked about Oracle Digital Assistant.

The bot recognized the request as a FAQ question and not a conversation and therefore displays the answer with no further ado. At the end of the answer, the bot displays two buttons for the user to rate whether the answer helped or not. Shown in the image below, the user does not want to provide feedback and instead asks a question about the cost of Oracle Digital Assistant.

Again, the bot recognizes this to be a question and provides an answer. Like before, the answer is followed with an option for the user to rate the quality of the answer.

This time the user decides that the answer did not help and presses the (n)o button (as indicated, the user could also juts have typed "n" to select the no button (without the quotes). Because the user selected no, the bot displays another pair of buttons asking whether the user wants to provide feedback to improve the answer. The user can now select one of the buttons, or ask another question.

If pressing (y)es, or typing y, the bot continues with the feedback prompt,

How The Sample Works

The sample follows the idea of Oracle Digital Assistant answer intents and uses Trainer TM as the model. Each answer is represented by an intent, as shown in the image below. In the image below, you see intents being created for conversations and answers. All FAQ intents have a leading prefix "qna.", which is not required but makes it easy to tell which intents are for conversations and which are for FAQ.

The image below has an FAQ answer qna.Pricing defined. The training utterances (Examples) are possible way for users to request the answer to be displayed by a user. When writing those sample questions, make sure you avoid fill words like "please", "thank you" etc. as they don't provide meaning to the question. Ideally you do perform crowd sourcing for authoring questions, which means that you ask others who are not part of the development team building the FAQ to suggest questions.

Unlike answer intents in Oracle Digital Assistant, the Answer field is left blank. If you add content to the Answer intent field then the intent becomes an answer intent. For the feedback functionality explained in this article, this is not what you want.

Similar to answer intents, the text to be displayed for a FAQ is saved in a resource bundle. The use of resource bundles allow you to translate answers into multiple languages, but also keeps all answers in a single location for ease of maintenance and avoids the need of creating a dialog flow state for each answer. The message key of an entry in the resource bundles must match the intent name the answer is for. So, as shown in the image below, the message key for the pricing question is qna.Pricing because the intent name is qna.Pricing

Update January 2022: In the course of migrating this blog to a new platform, we lost the original sample skill. The provided sample skill does NOT use the custom component shown in the screen shots and is a YAML only solution. You can however improve this sample by replacing the "showAnswer" state with the custom component state shown in the screen shots in this blog article.

The dialog flow shown in the image below only uses two dialog flow states for resolving intents as FAQ intents and to display the answer and feedback options.

Lines 36 – 45

The System.Intent component resolves user messages to custom intents or the unresolvedIntent. Custom intents can be conversational, like the "Welcome" and "unresolvedIntent"intent in the sample, or considered FAQ questions. For the solution in this article to work, all All conversational intents must have a representation in the actions: transition. This means that all conversational intents must be mapped to a dialog flow state. So if your bot has a conversational intent sendMoney then the actions transition must list sendMoney and map it to a dialog flow state.

For FAQ intents, you do the opposite. This means that no FAQ intent should have a representation in the actions transition. All intents that are not listed as actions: in the System.Intent component are passed to the next: transition (line 45). The next transition points to a dialog flow state showQna. The showQnA state references a custom component that you can also download at the end of this article. The custom component provides the answer to a question and also handles the user feedback conversation.

Lines 50 – 59.

The QnAFeedback custom component exposes the following properties

qnaAnswer (required) – expects the answer to be displayed. The sample code below shows how the answer can be read from the resource bundle based on the name of the top-scoring intent. So if the top scoring intent is qna.Pricing, then the answer read from the resource bundle is about pricing.
showFeedbackPrompt (optional) – allows you to display the feedback controls. If not set, then no controls will be displayed. The parameter supports Apache FreeMarker expressions so that you can conditionally show or hide the controls. The configuration in line 53 will always display the feedback option.
intentName (required) – allows you to pass information about the intent (answer) that a user rates. This information too then can be passed to a backend service by the custom component.
messagePromptsAndLabels (optional) – expects a an object with a specific set of key/value pairs to customize the feedback control labels and prompts (later more). In this example, the custom prompts and labels too are read from a resource bundle.

Passing Custom Labels and Prompts To The Custom Component

The image below shows the customization of the feedback control prompts and labels. A System.SetVariable component is used to create an object that contains key / value pairs. The values are read from the resource bundle, which means that they too can be translated. The following keys are used in the custom component:

componentPrompt – the prompt displayed after an answer along with a yes/no button
helpfulRatingThankYouMessage – is displayed when a user presses the (y)es button to indicate that the answer helped
yesButtonLabel – label used on all "yes" buttons
yesButtonAccessKeywords – a comma separated string of shortcuts to select a button. For the yes button this could be a string like: Y,y,YES,Yes,yep,sure
noButtonLabel – label used on all "no" buttons
noButtonAccessKeywords – a comma separated string of shortcuts to select a button. For the no button this could be a string like: N,n,No,NO,Nope,nope
noUserCommentThankYouMessage – message displayed when the user decides to not providing feedback by pressing the "no" button
userCommentThankYouMessage = message displayed when user provided feedback
userCommentPrompt = message prompt displayed for the user to enter free text feedback

The prompt/label object needs to be passed as the messagePromptsAndLabels custom component property. If you don't set the parameter then default strings provided by the component are used. If your object misses a key, then also a default value is used.

Update January 2022: You need to add the state shown below to the provided sample skill. Its not yet contained

About the Custom Component

The custom component that you can download for this article is provided as is. It is not an Oracle product but considered a sample. Free feel to use the code and extend it to your needs. As shown in the image below, you need to customize the custom component code to make it call a backend service for collecting the user feedback.

If you want to use this sample in your Oracle Digital Assistant skills, then search for the /* !!! – or – TODO strings and replace them with calls to a backend service. The backend service can then save the user feedback or pass it to an agent responsible for retraining the skill. You could, for example, use the Oracle Rest Data Service (ORDS) as a backend service to write user feedback to an Oracle database. You could then use Oracle Visual Builder Cloud Service to display the user feedback to your bot admins.

About Tokens Used

To maintain its internal state, the custom component defines and uses tokens as shown below. You can change the token values if you like. They need to be unique throughout a digital assistant (not just a skill) though

const feedbackComponentStateVariable = "_ZmVlZGJhY2tDb21wb25lbnRTdGF0ZVZhcmlhYmxl";
const feedbackComponentPostbackToken = "_ZmVlZGJhY2tDb21wb25lbnRQb3N0YmFja1Rva2Vu"
const helpfulPostbackToken = "_IGhlbHBmdWxQb3N0YmFja1Rva2Vu"
const addCommentPostbackToken = "_ZmVlZGJhY2tQb3N0YmFja1Rva2Vu"

Avoiding A Channel Lock-in

The solution works across channels and not just for the web. This means that you can also test it in the skill conversation tester.

Note: The sample skill uses HTML markup in the answers which does not render well on other messengers. If you plan using your skill on different messengers (not just the Oracle Web channel) then you need to delete the markup (including links) or, as all answers are read from a resource bundle, consider creating channel specific answers by defining keys as <intent name>_<channel type>: E.g. qna.Prices_web, qna.Prices_facebook. The expression you need to reference from the custom component qnaAnswer property would then be:

${rb(iResult.value.intentMatches.summary[0].intent)}_${system.channelType}

With the above change to the provided sample skill you can build skills with a FAQ implementation that truly is channel agnostic and that does not lock you into e.g. the web channel.

Note that using channel specific resource strings to me sounds like the best option to support channels that support markup and those that don't.

When To Use Oracle Digital Assistant Answer Intents

Built-in solutions should always be considered before building custom solutions.

Unless you require user feedback to an answer, please consider using the Oracle Digital Assistant answer intents. Using Oracle Digital Assistant answer intents ensures your FAQs to benefit from future feature updates and improvements added to answer intents. As a possible strategy, you could consider using both because you can run the solution explained in this article parallel to using Oracle Digital Assistant answer intents. For new answers added, you could use the feedback option explained in this article until you consider an answer to be mature, which is when you turn it into a built-in answer intent.

Sample Downloads

Below are the links for you to download the sample skill used to create the screenshots for this article as well as a link to download the generic feedback component.

Sample Skill
Update January 2022: In the course of migrating this blog to a new platform, we lost the original sample skill. The provided sample skill does NOT use the custom component shown in the screen shots and is a YAML only solution. You can however improve this sample by replacing the "showAnswer" state with the custom component state shown in the screen shots in this blog article.

The custom component that can do this is provided in the link below.

generic feedback component

Author

Frank Nimphius

TechExchange: How-to allow customers to provide feedback on the usefulness of answers to frequently asked questions

The Sample In Action