This article introduces a standard pattern that can be used to achieve the goal of obtaining all rows from a REST endpoint when that endpoint only supports a paging API. What do I mean by a paging API? Well that's the case where there is no option to just ask the endpoint for all rows at once but instead you have to pass it (or it will assume) some fixed number of rows to fetch per call. An example is the pattern used by Visual Builder Business Objects and ADF Business Objects exposed through REST that have a URL pattern like this: <some endpoint>?limit=25&offset=0. Meaning start at row zero and give me 25 rows. The return payload will then typically include a flag e.g. hasMore, which tells you that there are more rows available, in which case you would need to issue another call to get the next batch of rows, starting from where you last left off e.g. <some endpoint>?limit=25&offset=25 and so on, until hasMore=false.
Of course REST APIs differ and if the endpoint has a flag to ask for all the rows at once then just use that, but let's stick to how to implement this recursion for the core Visual Builder Business Object case.
For the sake of illustration I'll be using a Business Object called Participant with an id field and an email field and my final goal is to fetch all of the emails into a single array variable in the page.
To keep your code readable and maintainable, I strongly encourage the liberal use of well defined types to make mapping clearer and less error prone. For my use case, the Participant Business object exposes a GET many endPoint for which we'll want to generate a type, and I'll also create a simple object for the array row type called emailRecordType:
I'll need somewhere to store all the data, so I create a page variable called emailArray with an Array type and an Array Item Type of the emailRecordType defined above. Of course this variable will initially be empty.
As the title of the article mentioned, the process of making multiple calls to the endpoint to get all the rows will be managed through recursion, e.g. I'll have an action chain that will call itself, essentially looping until all the rows (or some max number of rows) have been obtained. So the first part of this will be to define the API for that Action Chain in terms of the inputs to it and the data that it will return.
So I create a new Action Chain called fetchBatch and define input parameters and a return type as follows:
Some details on each of those input parameters:
The return type for the Action Chain is again an array of the emailRecordType I defined in step 1.
Now we need to implement the Action Chain and the first step is the actual call to get a batch of records. To do this. we just drop a Call REST Endpoint action into the flow and wire that to the endpoint that I'm getting the data from:
And let's look at that assignment mapping in more detail:
You can see here that the fetchBatchSize and the startFetchFrom input parameters are mapped to the limit and offset parameters on the REST call. In this case I've also set onlyData (to true) and the fields that I want in the return payload to optimize things a little, but that's optional.
The return type for the REST action is the getall_Participant type that was set up in step 1.
The next action to add to the chain is one to assign the returned email addresses into the accumulator array. This just uses an Assign Variables action with the following mapping:
Notice how the Reset Target for the array assignment is set to none. This is important otherwise the accumulator won't do it's job!
Next we drop an If action into the Action Chain. This will do two things for us:
The actual condition used in the IF is:
[[ $chain.results.callRestChunked.body.hasMore && $chain.variables.accumulator.length < $chain.variables.maxRows ]]
If the recursion check passes then this is the point at which we need to recursively call the same action chain. To do this, we drop a Call Action Chain action in and map it to use this same chain. The important part of course is is the mapping of the parameters:
It's all pretty simple 1-1 mapping except for the value passed into startFetchFrom which of course needs the incremented fetch starting position which we can calculate as shown here by just adding the batch size to the previous start point.
Recursively calling the Action Chain will of course result in additional rows being added to the accumulator, so on our way back up the recursion stack we need to pick up that update. So we drop another Assign Variables action in after the Call Action Chain and re-assign the output of the recursive call back to the accumulator.
Important! Notice that this time we set Reset Target to empty. If this was not done, the final array would include duplicated data (Just think about the logic of the recursion for a second and you'll get it).
With that we're all done. All we need to do now is return the accumulator array to the caller, so we drop a return action in and do so:
And here's an overview of the whole chain that has just been defined:
Once this chain is defined you can then just call it from your desired action chain and the recursion will take care of the rest., for example: