TechExchange: Strategies for dealing with slow API responses
March 24, 20228 minute read
Article by Vincent Heinink, March 24 2022
Oracle Digital Assistant can call backend APIs with custom components. The time these APIs take plays a role in user experience. If there's no way to improve those API response times, then you need to think about how you're going about user experience and acceptance. This article shows how to provide periodic feedback to users so that time no longer matters.
These are some strategies you might want to consider:
1. If you’re using the Oracle SDKs for Web, iOS or Android, you can use a new 22.02 feature to for adding wait messages when there are delays in bot’s responses.
2. Handle this as an asynchronous process – you might be able to update the user that there request is being processed and they will receive an update via Slack/Teams/SMS/Email (first three would be good candidates to use Oracle Digital Assistant Application-Initiated Conversations feature).
3. If you’re not using one of the Oracle SDKs and you need to respond in a synchronous manner, you could have the custom component periodically update the user on progress by sending a customized wait message. This is the focus of the rest of the article.
Next to the actual use-case this article also shows how one can use OCI services from within a custom component, a convenient way of setting up required OCI resources using OCI Resource Manager/Terraform and a testing strategy for custom components.
Let’s start with a video showing the problem and solution:
Architecture
The user sends a message to ODA that prompts the retrieval of the quote of the day. The ODA dialog flow invokes an ODA custom component multiple times with the initial call (activation is marked as read in below sequence diagram) continuing to run in the background but sending the initial wait message to the dialog flow.
The subsequent custom component invocations (marked in green) check for results in a cache and either send a subsequent wait message or the final response.
For the custom component to check for API responses across multiple invocations that could be spread across multiple custom component containers (replicas of the same custom component service) it uses a NoSQL Database on Oracle Cloud Infrastructure.
Skill
The skill shown in above video is available here. The main parts of the dialog flow are where it invokes the custom component:
Which properties reference the skill custom parameters (under settings – configuration):
Notes:
1. When importing the skill the da.ociKey custom parameter (which is of type secure) will be empty which will cause the skill to not work. If you don’t want to setup the OCI services and just see the skill running you can provide a dummy value for it.
2. The custom component is getting its private key and other information for accessing OCI services through these custom parameters. If the custom component would be hosted on OCI functions, an alternative would be to use resource principal. This is left as an exercise to the reader.
And then where it shows the quote:
Custom Component
The custom component package is part of the skill archive and can be downloaded from the skills custom component services after importing the skill.
The custom component package has three implementations:
1. QuoteOfTheDay.Simple.js – No wait message implementation.
2. QuoteOfTheDay.ProgressIndicator.js – Wait message implementation using local in memory cache.
3. QuoteOfTheDay.ProgressIndicatorExternalCache.js – Wait message implementation using external NoSQL table as cache.
Going over the implementation of #2 will help in understanding #3.
1. First invocation races two promises, the API returning a response quickly and a timer for the interval at which we want to send wait messages to the end-user:
2. If the API response is slow, asynchronously the cache will be updated:
3. Subsequent custom component invocations check the cache periodically and wait for the response to be available in cache, the API timeout or time to update user via wait message:
Custom Component Testing
To ensure custom components – continue to – work. They are tested with a Postman collection with tests. These unit tests work without Oracle Digital Assistant dialog flow. One simply starts the custom component locally and runs the collection through a Runner in Postman.
Running custom components locally is documented here
For the custom component to check for API responses across multiple invocations that could be spread across multiple custom component containers (replicas of the same custom component service) it uses NoSQL Database on Oracle Cloud Infrastructure.
Note: There is potential cost involved with creating OCI resources.
Setup
To make setup easy, a Terraform script is provided. OCI Resource Manager allows you to automate the process of provisioning with this Terraform scripts (more in documentation). The provided Terraform script creates the NoSQL Database table with minimal capacity in a separate compartment, creates a user, generates an API key for the user to login and creates a policy for this user to only allow access to the NoSQL Database table.
Setup steps:
1. Login to the OCI Console and make sure you have required privileges and service limits.
2. Search for or navigate to Stacks under the Resource Manager category
3. Create Stack
4. Upload the QuoteOfTheDayStack.zipzip file, Name the stack, optionally select a compartment to create the stack in (stack itself will create a new compartment inside of the compartment you select – so you could go with root compartment for the stack), Next
5. Enter the region you want to provision in (making sure you have subscribed to it), Next
6. Review, check the Run Apply, then Create
7. The stack will now be created and applied in your tenancy. The output tab of this job run shows the information ODA’s custom component requires for accessing the NoSQL Database.
8. These outputs are used to update the skill custom parameters to allow the skill access. Pay special attention to copy the complete ociKey (including —–BEGIN RSA… and PRIVATE KEY—–). See Skill section for more details.
Tear-down
One you no longer need the infrastructure you can destroy the stack (resources are then freed up) and delete the stack (removes the definition). Please make sure you don’t delete the stack before destroying it as that would leave you cleaning up individual resources by hand.
Destroying the stack; press the Destroy button and confirm:
Deleting the stack; Select Delete Stack from More Actions drop down and confirm:
Summary
The time taken by APIs plays a role in the user experience. One of the ways to handle slow API responses is by having the custom component update the user with wait messages. This article has shown a scalable approach by using NoSQL Database on OCI infrastructure as a cache.
