article by Frank Nimphius, March 2023

Visual Flow Designer in Oracle Digital Assistant has changed the way skill developers enable users to upload documents, audio, video and images within a conversation. This article covers the two most common conversational use cases, which are

• digital assistant users that are directed to a dialog flow state to upload a document or to skip the upload. An example for this use case is a expense reporting system where a user needs to upload a receipt for an expense after filling in all of the required expense reporting fields. If a use does not have a receipt, they would use an action item to suppress the prompt for upload
• digital assistant users that are directed to a state where they can either manally provide the expense information or upload a receipt from which an artificial intelligence (e.g one you find in Oracle Cloud Infrastructure AI (see OCI Vision)) extracts the required information. 

Note: This article covers how to add attachments within a conversation. It does not cover accessing attachments for processing them

Use case 1: Users are prompted to upload a file or to skip

In this first use case, a dialog flow state is setup to accept attachments. Users who cannot upload an attachment would use the cancel option (which you could re-name to "skip attachment" if you like). The image below shows the flow for this use case. A dialog flow state prompts users to upload an attachment. Two other states either display the payload created for the uploaded attachment or achknowledge the cancel operation.  

Below images show how to create a Common Response component state that is configured so it can be modified to handle attachment uploads. On a component, or the flow's start state, add the Add State context menu option to see the component state templates. In the component templates …

… select User Messaging, then Create Text Menu and finally …

… Create Action Menu. Name the new dialog flow state with a name mathing its role in the flow, e.g. PromptForAttachment

Select the PromptsForAttachment state to open the property inspector. Use the Create option to create a flow scope variable of type Map. Name the variable anyway you want, e.g. attachmentVar. The variable must be of type Map because the response to a document, image, audio or video getting uploaded is an object. 

Next, change the state configuration by pressing the Edit Response Items button. 

Change the existing metada with the metadata shown in the image below. Notice that it contains a prompt as well as a cancel button option

Note: The sample has the label strings directly added to the metadata. This is good for a quick sample but not for any production quality bot. In your production quality bots, please reference labels in resource bundles so they can be easier edited by a copy writer but also translated into different languages. 

To display the payload returned in response to the uploaded attachment, edit the printMessage state (a Send Message component state) with the expression shown below. 

When you run this sample in the conversation tester or a messenger (like the Oracle web messenger), the printed message shows a JSON object with attributes. The attribute your are mostly interested in is url . The URL points to the location of the uploaded attachment for your to pass it to a custom component for further processing.

To access the url from the payload, you need to change the expression as follows

${attachmentVar.value.url}}

Notice the ".value." that ensures that the payload is not returned as a string but an object.

Hint: The sample skill can be downloaded at the end of this article

Use case 2: Users can either upload a file or be prompted for the required information

This use case allows users to either manually add the information for an expense item or upload an image that then could be downloaded in a custom component and passed on to a service for further processing. If you start with an empty flow diagram, then choose the Add start state from the Start state as shown in the image below.

The implementation of the use case requires a composite bag entity and a simple enttity event handler (EEH). Both need to be created before you build the flow. So, from the state component templates you choose Resolve Composite Bag to create a new composite bag state in the flow diagram. Provide a meaningful name for the state (e.g., PromptForExpense) and …

select the state to open the property inspector. Use the Create button to create a new variable in flow scope and set its type to the name of the composite bag entity. 

Again, you need two additional states: One state that prints the information the user entered manually, and another state to show the payload of the uploaded attachment (at least this is what the skill does that you can download at the end of this article). In the example, the composite bag entity variable is named expenseItem. When you type that name in the message field, an expression builder gets triggered providing you with code insight help. The ReceiptImage bag item is of type attchment and would hold the payload of any uploaded document. 

To obtain the URL of the uploaded document, the expression you use would be ${expenseItem.value.ReceiptImage.url}

In the state where you simply print out what the user has entered, you use the following expression

The image below shows the composite bag entity for this sample. Notice that the first item is amount, which is of type CURRENCY (an entity), The ReceiptImage bag item is of type Attachment and has its out of order extraction set to Always. This way, when a user uploads a document instead of providing the amount, the ReceiptImage value gets slotted.

This also means that the prompt that is set on the Amount bag item asks the user to either provide an amout or to upload a receipt for the expense (again I recommend you use references to resource bundles to add the prompt)

The part that entity event handler are used for is to detect whether the ReceiptImage bag item got a value, and if so to skip the other bag items from prompting. To create a entity event handler, press the + Event Handler button and provide a name for the deployment package as well as a name for the event handler.

Hint: When I create event handlers, I always make sure the name of the event handler mirrors the name of the composite bag entity it is for.

The image below show the code used for this example, The resolved function on the entity level is used to return attachmentFlow as an action transition when an image was uploaded. If the user provided expense item values manually, then the next transition would be followed.

The item level code uses the shouldPrompt function on each bag item to return false when the ReceiptImage bag item got a value. This way, no prompt is displayed when the user uploaded an attachment. 

You can test the composite bag entity in the conversation tester. The image below uses an attachment saved on Pixabay. To add it when prompted for the amount, you press the Attach button next to the message field and provide the URL to an image.

When you then submit the attachment and navigate to the variables view in the conversation tester, you can have a look at the JSON object that gets returned as a payload.

The payload may be different for different messenger types. Below is the same example displayed in the Oracle Web SDK. 

When prompted for the amount, select the attachment icon to open the upload dialog. This time I selected the dog image from my local machine …

… to be uploaded to Oracle Digital Assistant. As you can see, the image is temporarily tored in ODA and a url is returned within the response payload. The URL can be accessedd from a custom component or entity event handler, which you would create to further process the image.

In the use case of an expense item, processing could be to send the image with the receipt to an AI service to read it and return the required information for the expense. 

If a user does not upload an attachment but manually fills in the expense details, the navgation follows the next transition, which in the sample is to a state that prints a message with the entered values.  

Downloads

All samples can be tried in the embedded conversation tester or with a messenger like Oracle Web SDK. To start the samples, simply type any message (e.g. hello).

Skill using attachment state (use case 1)

Skill using composite bag entity (use case 2)