article by Frank Nimphius, February 2020
Bot conversations in Oracle Digital Assistant are not sequential, or in other words, many paths lead to the same result. The image below shows examples of user input and the expected outcome. Notice that "Please show me the menu", "I like to order pasta". I like to order a pasta with bacon" and "I like to order a pasta with bacon and garlic" lead to the same outcome, which is the confirmation of an order. Where the different user messages differ is in the number of states that are visited in the context of the conversation. To throughly test a skill in Oracle Digital Assistant, all possible conversation paths must be tried, and this for every change you apply to the dialog flow or any version or clone of a skills.
Good news is that Oracle Digital Assistant introduced the first implementation of a test suite that allows you, on a skill level, to record a conversation in the embedded conversation tester, which then can be repeatedly run whenever needed.
With this you can say that a conversation is the unit of testing in Oracle Digital Assistant skills.
“Tell me and I will forget, show me and I may remember; involve me and I will understand.”
― Confucius
This article provides a starter skill, as well as simple steps you can follow to explore the test suite feature and learn how to use it.
To follow along, please download download the starter skill (see link at end of article) and ensure you have Oracle Digital Assistant 20.01 or later available. In Oracle Digital Assistant, import and open the started skill (if you don't see the import, use the search field and type "Alfredo".
Before You Start
The test suite feature is not enabled by default. To enable it, in Oracle Digital Assistant. select the "burger" menu item and choose the Feature Management option in the Setting menu section. Change the profile list to Enable all and apply the changes.
Exploring the Test Suite
Navigate to the Pasta Alfredo skill and open it. Press the conversation tester to launch the embedded skill tester. Notice (see image below) that the tester header contains links to access test cases and test results, as well as a link to create test cases by saving a conversation.
Start by typing I like to order pasta into the Message field and press the enter-key on your keyboard.
You can navigate the card layout to see the pasta menu. To continue, type I want bacon pasta into the Message field and press the enter-key. It doesn't matter what the message text is for as long as it contains bacon pasta (the magic of entity extraction).
Note: Don't press the buy button. by the time of writing, due to a know issue, using a postback action results in an invalid test case that wont be saved.
Next, type Cheese and garlic and press the enter-key.
In the order confirmation, notice the the date and time information. This will become relevant later when you test the conversation. Next, click the Save as Test Case button. Provide a name like shown in the image below and press the Save Conversation button.
Reset the conversation by clicking on the Reset link in the header. Next, type I like to order bacon pasta and beef ragout. Notice in the image below that this user message displays a disambiguation dialog alerting the user that only one pasta can be ordered at a time. This dialog wasn't displayed in the first conversation. You can navigate the cards and notice that the card layout only contains two cards.
Next, type Okay. If I can only have one, then beef ragout please. This message has "beef ragout" as a valid entity value in it (which is all that matters)
Next, type I like cheese and oil into the Message window and press the enter-key.
After the confirmation is displayed, click the Save as Text Case link and save the conversation as shown in the image below.
Important Note: Test case don't need to be complete conversations that end with a return statement (transition). You can, for example, stop the conversation in the tester at any time and save the test case. The last message displayed by the bot will then be the outcome the test case will determine success or failure with.
Next, select the Test Cases link. Notice that there are 4 test cases. Two of the test cases is what you created when following the instructions in this article. Two test cases were part of the starter skill you imported. What you learn with this is that test cases you create belong to a skill and will be exported and imported with the skill. This also means that when you version or clone a skill, that those skills will have contain the test cases too.
Note: Test cases can also be exported for a skill without exporting the whole skill. For this, you would exit the skill (navigating to the skill dashboard) and then use the menu icon in the right bottom of the skill tile. This way you can export test cases to import into other skills.
Notice that the conversation recorded for each skill is saved as a JSON payload. You may want to compare the payload of one of the test cases you created with the SMS test case already provided. You will notice differences in the component rendering expressed in the payload. So for a test case it also matters which channel simulation was used when recording the conversation.
Next, press the Run All Test Cases button to execute the tests.
You can change the name of the job if you want. For this article, just keep it and press Submit.
Give it a few minutes to process. Then, click the Test Run Results link.
Notice in the image below (and un your test run) that two test cases out of 4 failed.
Expand one of the failed test cases to learn about the problem. The problem in this case is the date and time information printed within the order confirmation (told you that this will become a problem). Reality is that you cannot exclude that content in a bot message changes. If it wasn't the date and time, it could be a generated order ID that changes between bot test runs.
One option to fix the problem would be to press the Apply Actual button to modify the test case. While this helps for infrequent changes, it does not for the date and time in the example. To solve this problem, and to fix the test case forever, click on the test case name link.
The link navigates the UI to the test case definition. Scroll the Conversation field content to the bottom. Find the CONFIRMATION entry with the date and time information.
Replace the date and time information with a place holder. The image below uses ${DATE} as a placeholder. You can however choose whatever name you want for the placeholder. Notice that when you move the cursor out of the Conversation field, the DATE placeholder is shown as a variable.
Next (inportant!), do the same for the other test case that failed, The test case too contains the date and time information.
When you rerun the test cases with the applied changes, then all tests succeed. When you switch to the test case results and you see tests still in progress, refresh the browser page frequently until you see the final test results.
Summary
This article provided a brief overview of the test suite in Oracle Digital Assistant that you can use to record and replay conversations. The test cases are exported, versioned and cloned with the skills. You run the test cases from the conversation tester, or from anywhere in the skill editor using the Run Tests button.
Note that a test case stops at the first difference in a conversation. If there are 3 user interactions in a conversation and the first one fails, then the test case stops here and does not continue testing the other two interactions. If you want to test different parts of a conversation it is recommended to create multiple shorter test cases and e.g. use entity slotting to skip some states in between the conversation.
Download
Related Content
TechExchange Quick-Tip: How-to Test Apache FreeMarker Expressions in Oracle Digital Assistant
All 2-Minute Oracle Digital Assistant TechTip Videos on YouTube
