External Data Engines I
By Tim Dexter on May 13, 2008
Well it's almost mid May here in Colorado, so you know what that means, yep, its snowing! Not too heavily but the temperature has dropped from a balmy 70F yesterday to a chilly mid 30s today. Apparently we need it, inspite of nearly a 120% of average snowpack for the year we are facing drought conditions in Colorado this summer - so we are xeriscaping our garden. My wife and I are enjoying a discussion around me wanting to rip up the water hungry grassy turf we have in our yard and replacing it with concrete ... Im kidding. Anyhoo, enough 'weather' related musings ... remember Im English, we are nearly all obssessed with it.
Most of you that use the BIP Enterprise release know that it can get data from almost anywhere, files, dbs, web services even Excel with a little help. I have had a few questions about the 'how' on getting 'remote', non-database data i.e. a 'data engine' already exists and it serving up XML. I thought I would cover some of those and end on the big question that I have been working on with a customer. What about a data engine that BIP can not connect directly to? This tied in with, 'we dont want to use your scheduling engine we have our own, how can it work with BIP?' Tasty questions, we'll come back to later.
So, what are the options for fetching data from an existing data engine? Thats going to depend on how connected your engine is to your network. As I mentioned above BIP can connect to data sources in multiple ways.
HTTP/S - if your engine can serve up data upon an HTTP request, then BIP can call it and pass parameters on the URL. If you can write a servlet that can call the engine and serve up the data on its behalf you could go down that route too.
Web Services - can you wrap your engine in a web service? Again, BIP can call for the data via the service and pass parameters into the service to influence the data set.
XML File - this is probably the simplest method. IF your engine can generate a file and drop it into an accessible directly for BIP to pick up then you have a solution. Its a simple case of registering the directory with BIP and then setting up the filename in the report definition for BIP to look for.
Excel - this needs a little effort to get the data from the file but we provide the APIs you just need the servlet to read it and serve up the XML data to BIP upon request.
All of these options will require the BIP report to be scheduled to pull the data periodically. In the first two cases thats probably OK, BIP is making an indirect call to the data engine requesting data at the time the report runs. In the last two cases BIP is going to rely on the files being refreshed prior to the scheduled report running otherwise you are going to see the same data.
How can we get away from this 'pull' model where a report is scheduled to pick up the data, what if we wanted to control the timing of the report, say those XML files were being dropped into a directory periodically and each time the new file was dropped in we wanted to get BIP to run the report. While we are at it, maybe we have retail branch sales data coming into a central server, all of those XML files have the same data structure. We do not want to define a report for each branch, we only want a single report definition that picks up the appropriate filename at runtime, processes it and sends an email to the branch manager with their results and a copy to corporate. I could go on, but thats enough of a scenario for the time being. Next time I'll dig in and talk about how you might tackle just such a scenario using BIP.