"Large scale data management" for Financial Services
By ambreesh on Jul 13, 2011
The problem of managing large amounts of data - structured data for this writeup - is pervasive in the financial industry. Compliance, Risk, Analytics, Pricing etc. all require ingesting, cleansing, transforming, standardizing, aggregating, persisting, analyzing and reporting on very large quantities of data. Given Oracle's pedigree in data management, I don't think it would be a surprise to you that we have a large set of technologies that help our FSI customers with their data management issues. We are also taking these technologies to our partners and helping them achieve enterprise class scale and reliability for their applications using this "large scale data management" platform.
The following diagram shows the relevant technologies that collectively we call the LSDM platform.
The baseline is Exadata or large SPARC systems. Data movement technologies, GoldenGate and ODI are the layer above - the combination allowing users to move data from a database instance to another instance in near realtime (change data capture only) and manipulating it at the destination. The destination instance is 11g, and the Spatial option brings in Semantics into the equation. Semantic data stores are starting to become popular in the FSI, since modeling of complex and continuously morphing relationships is easier using semantics than relational databases. For an industry that builds products that are so intricate that most personnel - and machines - have no clue about the component parts of these products, semantics will likely be mandated by the regulators.
Above the database are the in-memory data technologies that can be used in a variety of ways – Coherence as the in-memory data grid, and TimesTen as the in-memory database. These technologies are essentially performance related technologies. Think of Coherence as a large data cache where the data is dynamically provisioned across memory resources of various servers, and compute can be shipped to these servers – moving the compute to the data which is typically faster than the other way around. TimesTen is a in-memory SQL database, which can be linked with the compute making the two be a part of the same address space, again accelerating performance.
And the BI layer sits above all these technologies, helping with analytics and reporting.
We are positioning this stack with our customers and our ISV partners, in the reference data and risk space. Both these areas need large volumes of data to be processed, and the ISVs that we have spoken with are excited about working with us.