Data Replication for Stores
By David Dorf on Feb 18, 2010
A tough problem for store systems is reliably moving data between the home office and stores in both directions. There are many solutions to this problem, each with its own positives and negatives. One solution is using the open-source software SymmetricDS, which was created for this very problem during the Oracle Retail POS implementation at BigLots. The original developers formed JumpMind, a company specializing in helping retailers use open-source to solve difficult problems. I recently spoke with Chris Henson, the co-founder of JumpMind about SymmetricDS:What does SymmetricDS do for retailers?
SymmetricDS is an open-source, web-enabled, database independent, data synchronization software application. It uses web and database technologies to replicate tables between relational databases in near real time. SymmetricDS is database agnostic and can even synchronize data between different types of databases. For retailers, it provides a proven, consistent, simple, and flexible way to move business-critical data between a central enterprise database and multiple store-located databases. For example, the software has been used to move data that originates from corporate systems, such as item, tax, inventory, and employee data to stores. It has also been used to trickle back sales and inventory data from stores at near real time.
SymmetricDS has been proven to withstand the intermittent network and low bandwidth conditions sometimes found in the rugged wilderness of retail systems. Additional database tables can be added to synchronization profiles in real time with a simple change of SymmetricDS configuration by inserting into the central enterprise database.
Where did the idea for SymmetricDS originate?
The idea of SymmetricDS came from the need to synchronize many small, inexpensive databases at stores to a single, large, more expensive one at a corporate location. We needed something that would trickle sales data over a slow network and survive outages. Some of the data needed to be filtered horizontally as subsets, so each store had its own data. We also had sensitive credit card data that needed to be filtered vertically and re-routed to a separate, secure database. The initial implementation was for a retailer who was expanding, so the solution needed to be able to scale, and it needed to be easy to bring online for new stores.
How has SymmetricDS helped Abercrombie & Fitch?
SymmetricDS provided Abercrombie & Fitch the ability to quickly and efficiently integrate data targeted for and sourced from their store's Oracle Retail POS databases. It allowed data to be reliably routed to specific locations based on store numbers in the data itself. SymmetricDS also was pivotal in a switch in direction from the typical Oracle Retail store server architecture to a regional web-service-based architecture.
Not only is SymmetricDS used to synchronize core Oracle Retail tables, but it is also used to synchronize and route custom tables that contain country, brand and store specific parameters, clock punches and employee schedule data. The software also plays a key part in the infrastructure that deploys software to the stores.
SymmetricDS's extension points were used to update original transactions in the corporate database during the trickle back of returns, and are also used to integrate sales data in XML format to Oracle Retail Sales Audit via JMS. In addition, because SymmetricDS is based on standard HTTP technologies, it has been secured using standard web practices to meet both PCI and general security best practices.
Is there strong support for SymmetricDS in the open-source community?
SymmetricDS has a thriving, global open-source community. We are a healthy project with active support in the open source community. Hosted on SourceForge, SymmetricDS ranks as a top 500 project. Our ratings section is all "thumbs up." The community drives us with constant feedback for features, fixes, and improvements. For example, when we first started the project, we supported two databases, Oracle and MySQL. After requests and help from our users, we now have support for nine databases. Our success stems from listening to our users and
responding, which produces something that people want and find useful.
We are very excited about the upcoming SymmetricDS 2.0 release. It is just a few weeks off.
The biggest demands that we hope to fulfill post-2.0 are:
1. Administration and Configuration User Interfaces.
2. Conflict resolution strategies for tables that may be updated in bi-directional use cases.
3. Standard data transformation extension points.