Now that Big Data is officially the next big thing - I wanted to write about my thoughts on how big Big Data really is as it relates to Public Sector. First of all, I am reasonably certain that various specialized intelligence agencies are already using Big Data technologies to mine data in ways they have never done - and they have been doing it for more than 3+ years. Outside of intelligence agencies, how does this map out to the broader public sector? How can this be used to better the life of the common man?.
First and foremost, I personally think that folks don't realize the potential for doing things with big data sets. This is mainly because we never had the luxury to THINK BIG until very recently. I see two catalysts that is now driving the Big Data movement
a) Continued drop in storage costs and more storage availability ( forgetting the Thailand floods for a moment!)
b) Continued increase in COMPUTE power that allows us to PARALLEL everything (borrowing Larry's terminology from OpenWorld 2011)
Now that we have the luxury to think BIG - what are the things that we can think BIG on.
For example, NOAA today collects information from it's buoys that are floating out there in Pacific, Atlantic and Indian oceans to make assessments about Tsunamis, earthquakes etc. With our increased capability to collect and process more data, I see that these type of prediction models approaching more accuracy - thereby directly saving lives.
Another example could be the data that is currently being collected by all the dreaded speeding cameras across our great country ( definitely not in love with those for sure). However, if we can somehow have these data collection points collect more info, we can predict transport flows much more correctly.
At an even more personal level,now we can throw sensors all around the house and have temperature control systems that can specifically turn on specific vents based on very localized information (may be!).
I frankly think there are applications that have not been thought about that will be thought about and written in the near future. I also see a day where there will be an App Store for Big Data apps- You buy the App, Rent an Big Data Infrastructure on the cloud to crunch whatever data you want to crunch and consume the results. Easier said than done-but for some reason I feel that world is not that far away.