Integrate OCI AI Services in Visual Builder applications

Oracle delivers a comprehensive AI portfolio integrated in its cloud applications. There are all sorts of available AI Services including a Digital Assistant, Speech and Language and Document Understanding and more. You can use these services directly from your OCI account but, it's a much more useful to create friendly web interfaces for them with Visual Builder applications.

In this post we'll walk through the steps to build a Visual Builder Application that will upload an image of an invoice, process it through AI Document Understanding and return a JSON object for you to use however you'd like. Here is a video demo of a completed app using the AI service built with the Redwood Guided Process pattern:

The highlevel components involved in the solution are:

Visual Builder app with the Camera component allowing you to take pictures of invoices or pick images from your local files.
Images are stores in the OCI Object Storage
VB app passes the image to the AI service for processing – results are stored in a file on Object Storage.
VB app reads the results and process them.

Prerequisites

An Oracle Cloud user with permissions to use
- Object Storage
- AI Services / Document Understanding
Add an API key to your user if you don't already have one
Follow the first 5 steps in the 'Prerequisites / User API key' section of this post https://blogs.oracle.com/vbcs/post/connect-oci-to-visual-builder
- Save the OCID for your User
- Save the data for your key
  - Key fingerprint
  - Private Key

Create an Object Storage Bucket

The Visual Builder application will work with two files. First, the app will upload an image of an invoice, either from the file system of your PC or from taking a picture with your phone. After the image is processed by the AI service, a JSON document will be produced.

You will need an OCI Object Storage bucket to hold these files.

In your Oracle Cloud console, under Storage, click "Buckets"
Select the compartment you want to work in
Click the "Create Bucket" button
Enter a "Bucket Name" (save this for later)
vb_files
Click "Create"
Click on the name of your bucket
Save your Namespace
Click on your Compartment
Save a copy of the Compartment OCID

Create a new Visual Builder application

In your Oracle Cloud console, under Developer Services, click "Visual Builder"
Click the name of the instance you want to work in
Click the "Service homepage" button
Create a New application
Enter an "Application Name"
ProcessInvoice
Click "Finish"

Create Service Connections

You will create two service connections, used to access the OCI REST endpoints for AI Documents and Object Storage.

Object Storage REST API

Open the "Services" tab, click the "+ Service Connection" button
Click "Define by Endpoint"
Set the Method to "PUT"
Follow this link to get the OCI Object Storage REST API URL for your region.

https://objectstorage.us-ashburn-1.oraclecloud.com
Enter the URL for your region and add "/n/{namespaceName}/b/{bucketName}/o/{objectName}"
https://objectstorage.us-ashburn-1.oraclecloud.com/n/{namespaceName}/b/{bucketName}/o/{objectName}
Set the Action Hint to "Create"
Click "Create Backend"

Configure a new Backend

Under Security, set Authentication to "Oracle Cloud Infrastructure API Signature 1.0"
Click the "Enter username and password" pencil icon
Enter your Key Id. This will be your "TenancyOCID/UserOCID/KeyFingerprint" found in the information you saved from the post linked above
The 3 pieces of data are separated with a "/". There should be no spaces in this string
Enter the private key that you saved from the post linked above
Click Save
Enter a Backend Name
oci_object_storage
Click "Next"

Finish Server Configuration

Enter a Service Name
oci_object_storage
Switch to the "Operation" tab
Enter an "Endpoint ID"
uploadFile
Click "Create"
Click "Finish"

Create an Endpoint to retrieve the results file created by the AI Document Processing service

Switch to the Endpoints Tab, click "+ Endpoint"
Set the Method to "GET"
Add "n/{namespaceName}/b/{bucketName}/o/result/{jobId}/{namespace_bucket}/results/{fileName}" to the URL
Change the Action Hinto to "Get One"
Set the Endpoint ID to "getJobResultsObject"
Switch to the "Request" tab, click "+ Static Query Parameter"
Enter the Name
httpResponseContentType
Enter the Value
application/json
Click "Save"
Click "Finish"

AI Document Processing Service connection

You can find your AI services region endpoint here. Copy the endpoint for your region to use below.

Open the "Services" tab, click the "+" then click "Service Connection"
Click "Define by Endpoint"
Set the Method to "POST"
Enter your AI services region endpoint followed by "/20221109/processorJobs" in the URL field
https://document.aiservice.us-ashburn-1.oci.oraclecloud.com/20221109/processorJobs
Change the Action Hint to "Create"
Click "Create Backend"
Follow the same steps as above to configure the Security
Enter the "Backend Name"
ai_documents
Click "Next"
Set the "Service Name"
ai_documents
Switch to the "Operation" tab
Enter the "Endpoint ID"
createDocProcessorJob
Click Create
Click Finish

New Web Interface

Now that you have the services configured, you will create a simple Web App with a button to upload an invoice and an image object to display it.

Switch to the Web Apps section, click "+ Web Application"
Enter an "Application Name"
ProccessInvoice
Click "Create"
Close the left menu to give you more space to work

Add Page Variables

You will create the following Page Variables
ID	Type	Default Value
bucketName	String	<Bucket name from above>
namespaceName	String	<Namespace name from above>
compartmentId	String	<Compartment OCID from above>
imageURL	String
holdImage	Any

Switch to the "Variables" tab
Click "+ Variable"
Enter the ID and Type for each of the variables in the above table
After the first four variables click "Create & New"
After the last one, click "Create"
Once the variables are created, click on each of the variables with a "Default Value" in the table above, and enter the values you saved earlier

Page Designer

Add an Image Object to display the uploaded file

Switch to the "Page Designer" and open the "Components" tab
Search for Image
Drag an Image object and drop it into the grid in the designer
In the Image Properties enter "oj-md-12 oj-flex-item" into the Class Field
Drag the size of the image to 6 columns
Switch to the Data tab and click the "Select Variable" arrow
Select Page / Variables / ImageURL

Camera (File Picker) Object

Add a Camera Object. This button will work as either a File Picker when used from a PC or it will open the camera on your phone and upload the picture you take.

In the "Components" tab, search for Camera
Drag the Camera object and drop it into the grid in the designer
Optional: This will place the Camera button beneath the Image. If you would prefer it to be above the image; in the Structure tab, drag the Grid Row containing the Camera to be above the Grid Row containing the image
In the Structure section, click the Button under Camera
In the Button Properties, change the label to "Take Photo"
In the Structure section, click Camera
In the Camera Properties, uncheck "Video"
Note: Some phones will open a file selector instead of the camera if both Video and Camera are checked.

Add an Event to the Camera Object.

Switch to the "Events" tab
Click "+ New Event"
Select "On 'Selected Files'"

Create an Action Chain

In this Action Chain you will be using the following actions from the Actions Drawer on the left.

JS Code
Assign Variable
Call REST
Fire Notification

Actions

Set the image variables

The image that was captured from the camera or uploaded from the file system is passed into the action chain using the files array. Use this image to set the holdImage and imageURL variables.

Drag "Assign Variable" from the action drawer to the action chain
In Variable, select Page / holdImage
In Value, select Action Chain / [] files / {} item[0]
Drag "JS Code" from the action drawer to the end of the action chain
In the code block enter
```
$page.variables.imageURL = URL.createObjectURL($page.variables.holdImage);
```
This variable is used as the source for the Image object .

Add a REST Call to upload the file to Object Storage

Drag "Call REST" from the action drawer to the end of the action chain
Next to Endpoint, click "Select"
Expand Service / oci_object_storage and click "PUT" then "Select"
Next to "Input Parameters" click "Assign"
Drag the following from Page / Variables on the left to Target Parameters on the right
- bucketName to bucketName
- namespaceName to namespaceName
- holdImage to objectName
Click on "objectName" under Target Parameters, and in the Expression editor on the bottom add '.name'
$page.variables.holdImage.name
Click "Save"
Next to Parameters click Assign
Under Sources / Page Variables, Drag holdImage onto body in Target / Parameters
Click "Save"
Next to Content Type, click the Select Variable arrow
Select Page / Variables / holdImage
add '.type' to the end of the value
[[ $page.variables.holdImage.type ]]
Set the ID to "uploadFile"
Set "Store Results In" to "uploadFileResult"

Error Handling Notification

This section will be reused below for each of the REST calls. The top notification is the "Error" notification the bottom is the "Success" notification. Replace the Summary and Message values for each section.

Drop a Fire Notification onto "Create Error Handler" of the Call REST action
Click on Fire Notification and set the "Error" notification values
- Summary – {{ "Upload File – Error" + uploadFileResult.body.code }}
- Message – {{ uploadFileResult.body.message }}
Drop Fire Notification onto "Create Branch" of the If action
Click on Fire Notification and set the "Success" notification values
- Summary – File uploaded to Object Storage
- Message – {{ $page.variables.holdImage.name }}
- Display Mode – transient
- Type – confirmation

Add a REST Call to start an AI Document Processing job

Drag "Call REST" from the action drawer to the end of the action chain
Next to Endpoint, click "Select"
Expand Services / ai_documents and click "POST" then "Select"
Click "{ } body"
At the bottom right of the window, select Expression then enter the following in the text area

{
  "processorConfig": {
    "processorType": "GENERAL",
    "features": [
      {
        "featureType": "KEY_VALUE_EXTRACTION"
      },
      {
        "featureType": "DOCUMENT_CLASSIFICATION",
        "maxResults": 1
      }
    ]
  },
  "inputLocation": {
    "sourceType": "OBJECT_STORAGE_LOCATIONS",
    "objectLocations": [
      {
        "bucketName": $page.variables.bucketName,
        "namespaceName": $page.variables.namespaceName,
        "objectName": $page.variables.holdImage.name
      }
    ]
  },
  "outputLocation": {
    "bucketName": $page.variables.bucketName,
    "namespaceName": $page.variables.namespaceName,
    "prefix": "result"
  },
  "compartmentId": $page.variables.compartmentId
}

Click "Save"
Set the ID to "createDocumentProcessorJob"
Set "Store Result In" to "createDocumentProcessorJobResult"

Error Handling Notification

Add Error Handling Notifications following the instructions in the above section using the following values.

Error Values

Summary – {{ "Create Document Processor Job – " + createDocumentProcessorJobResult.body.code }}
Message – {{ createDocumentProcessorJobResult.body.message }}

Success Values

Summary – Document Processing Job Submitted
Message – {{ createDocumentProcessorJobResult.body.lifecycleState }}
Display Mode – transient
Type – confirmation

Add a REST Call to download the results file from Object Storage

Drag "Call REST" from the action drawer to the end of the action chain
Next to Endpoint, click "Select"
Expand Service / oci_object_storage and click "GET" then "Select"
Next to "Input Parameters" click "Assign"
Drag the following from Page / Variables on the left to Target Parameters on the right
- bucketName to bucketName
- namespaceName to namespaceName
- namespaceName to namespace_bucket
  - In the expression editor add ' + "_"'
  - Drag bucketName to namespace_bucket
    The final expression will be
    $page.variables.namespaceName + "_" + $page.variables.bucketName
- holdImage to fileName then add '.name + ".json"'
  The final expression will be
  $page.variables.holdImage.name + ".json"
Drag Action Chain / createDocumentProcessorJobResult / body to jobId then add '.id'
The final expression will be
createDocumentProcessorJobResult.body.id
Click "Save"
Set the ID to "getResultsObject"
Set "Store Result In" to "getResultsObjectResult"

Error Handling Notification

Add Error Handling Notifications following the instructions in the above section using the following values.

Error Values

Summary – {{ "Get Results Object – " + getResultsObjectResult.body.code }}
Message – {{ getResultsObjectResult.body.message }}

Success Values

Summary – Get Document Processing Results Object
Message –

{{ getResultsObjectResult.body.detectedDocumentTypes[0].confidence + ' – ' + getResultsObjectResult.body.detectedDocumentTypes[0].documentType }}

Display Mode – transient
Type – confirmation

Add JS Code

The JSON object returned by AI Document Processing has a lot of information in it. Add the following JavaScript Code to extract just enough to show that everything worked.

Drag "JS Code" from the action drawer to the end of the action chain
In the code block enter

let processedDocInfo = {};

getResultsObjectResult.body.pages[0].documentFields.forEach((element) => {
  processedDocInfo[element.fieldLabel.name] = element.fieldValue.value;
});

Add Fire Notification

Add a final Notification to display the results

Drop a Fire Notification at the end of the Action Chain
Set the following values
- Summary – Processed JSON Object
- Message – {{ JSON.stringify(processedDocInfo) }}
- Type – confirmation

Results Notification

Process an Invoice

This application should work with most standard invoices you can find on the web.

Search for and download an invoice in an image format (.jpg, .png)
To keep the examples from getting to complex, this application is configured to only work with images.
Click the "Preview" button in the upper right corner of Visual Builder
When the Preview page opens, you should be able to open the same URL on your phone as well.
Click the "Take Photo" button
- If you're on your PC, select the invoice file
- If you're on your phone, use your camera to take a picture of the invoice

The notifications should fire at each step of the action chain. The AI processing step can take a few moments. At the end of the chain, a simplified JSON object will be shown. Compare the values with the invoice.

Once you're through experimenting, you will probably want to manually delete the files in your bucket.

Next Steps

At this point, you have the data from the document returned in a JSON object. You could:

Create a screen to validate and update the data
Push the validated data into a Business Object or a Fusion Application
Add routines to handle other types of pre-trained documents
Train your own model in the AI tool to handle custom documents
Modify the application to work with other file types such as a .pdf file
Examine the JSON file in your OS Bucket to see what other information is available
Add a DELETE endpoint to your Object Storage service to use in your action chain for deleting the files after they have been processed

Although the steps above are for working with Document Understanding, they can easily be adapted to work with the other AI services as needed.

Integrate OCI AI Services in Visual Builder applications

Prerequisites

Create an Object Storage Bucket

Create a new Visual Builder application