X

The Oracle NoSQL Database Blog covers all things Oracle NoSQL Database. On-Prem, Cloud and more.

Migrate MongoDB data to Oracle NoSQL Database

A Chandak
Principal Product Manager

Introduction

Migrating data from a database to another entail design decisions related to data consistency, downtime of applications, and the key design differences between the two databases. When it comes to migrating across NoSQL databases, the challenges are multifold considering the many flavors such as document, key value, wide column, graph, etc. available today. Oracle NoSQL Database makes it easy to migrate data from different data models be it document, key-value or wide column databases.

Oracle NoSQL Database is a truly multi-model NoSQL data store which allows the customer to choose between key-value, JSON or table representation of their data. It is shard local ACID and provides control at the operation level for consistency and durability. One can leverage the efficient Elastic Search integration for their full-text index needs which includes full-text indexes on JSON documents. One can store Large Objects in Oracle NoSQL Database. The database provides a rich and extremely efficient LOB API. The database allows you to start with the key-value model of storing data and then can easily migrate to the table model and take advantage of the table semantics and SQL like Query feature. If you need joins then the database also provides the customers to build their data model as Parent-Child as against modeling as large nested documents.

Export-Import utility

Oracle NoSQL Database includes an Export/Import utility that allows you to export and import Binary data files from/to NoSQL database. With the 19.1 release of the database, this utility has been further enhanced to process multiple data formats. The utility now supports export data from NoSQL database to data files in Binary or JSON format, and import data files in Binary, JSON or MongoDB JSON format to Oracle NoSQL database.

Data format – Binary

Export from Oracle NoSQL Database   

Import into Oracle NoSQL Database

Data format – JSON   

Export from Oracle NoSQL Database   

Import into Oracle NoSQL Database

Data format – MongoDB JSON       

 

Import into Oracle NoSQL Database

Understanding Source and Sink:

 A source in the Import utility is file-based which contains the data exported from a database. Let us consider we want to migrate data from MongoDB to Oracle NoSQL Database. Here all the data from MongoDB are exported in JSON format and stored in files. These files are identified as “source”.

A sink in the Import utility is always Oracle NoSQL Database. We have already identified MongoDB extracted JSON file as the source and we want to import this data into Oracle NoSQL Database which would be called as “sink”.

User Experience - Export:

Using the export utility, one can export all the data/metadata from the Oracle NoSQL Database store to local file system. Using the utility one can export:

  • Application created data
  • Schema definitions such as table definitions, Avro schemas, and the index definitions.
  • TTL of every table record

The utility creates a package which contains data, schema, and logging information in a well-defined format which can be consumed by the import package.

The syntax for export:

# Export the entire kvstore or a given table to an export store. Use config file to specify export store parameters.

 java -jar KVHOME/lib/kvtool.jar export

             -export-all | -table table_names | -namespace namespaces

             -store storeName

             -helper-hosts helper_hosts

             -config config_file_name

             [-format BINARY | JSON]

             [-username user]

             [-security security-file-path]

             [-verbose]

User Experience – Import:

Using the import utility, one can import the data from local file system) to Oracle NoSQL Database.  Using the utility one can import:

  • Import MongoDB JSON format exported data into Oracle NoSQL Database
  • All the schema definitions and the data (table, Avro and none format data) from the backed up store to the Oracle NoSQL Database store. The utility first imports all the schema definitions and then imports the user data into the Oracle NoSQL Database.
  • Individual table or tables with specified namespace
  • The TTL of the record

The syntax for import:

# To import the schema definitions and data from Oracle NoSQL store into local filesystem:

java -jar KVHOME/lib/kvtool.jar import

        -import-all | -table table_names | -namespace namespaces | -external

        -store storeName

        -helper-hosts helper_hosts

        -config config_file_name

        [-status status_file]

        [-format BINARY | JSON | MONGODB_JSON]

        [-username user]

        [-security security-file-path]

        [-verbose]

Using Export-Import:

While examples can also be found in the official documentation page Using Export Import Utility.

For this article let us consider an existing Vacation booking application that stores Country information in MongoDB and decides to migrate its data to the enterprise scale Oracle NoSQL Database.

The country information is stored as JSON document and contains information such as – landlocked, shares borders with, latitude and longitude, currency, official name, official languages spoken etc. and more such information that might be useful for a person trying to book a vacation to a desired country. A sample JSON document is given below:

{

            "_id": {

                        "$oid": "55a0f42f20a4d760b5fc3153"

            },

            "altSpellings": ["US", "USA", "United States of America"],

            "area": 9.37261e+06,

            "borders": ["CAN", "MEX"],

            "callingCode": ["1"],

            "capital": "Washington D.C.",

            "cca2": "US",

            "cca3": "USA",

            "ccn3": "840",

            "cioc": "USA",

            "currency": ["USD", "USN", "USS"],

            "demonym": "American",

            "landlocked": false,

            "languages": {

                        "eng": "English"

            },

            "latlng": [38, -97],

            "name": {

                        "common": "United States",

                        "native": {

                                    "eng": {

                                                "common": "United States",

                                                "official": "United States of America"

                                    }

                        },

                        "official": "United States of America"

            },

            "region": "Americas",

            "subregion": "Northern America",

            "tld": [".us"],

            "translations": {

                        "deu": {

                                    "common": "Vereinigte Staaten von Amerika",

                                    "official": "Vereinigte Staaten von Amerika"

                        },

                        "fin": {

                                    "common": "Yhdysvallat",

                                    "official": "Amerikan yhdysvallat"

                        },

                        "fra": {

                                    "common": "États-Unis",

                                    "official": "Les états-unis d'Amérique"

                        },

                        "hrv": {

                                    "common": "Sjedinjene Američke Države",

                                    "official": "Sjedinjene Države Amerike"

                        },

                        "ita": {

                                    "common": "Stati Uniti D'America",

                                    "official": "Stati Uniti d'America"

                        },

                        "jpn": {

                                    "common": "アメリカ合衆国",

                                    "official": "アメリカ合衆国"

                        },

                        "nld": {

                                    "common": "Verenigde Staten",

                                    "official": "Verenigde Staten van Amerika"

                        },

                        "por": {

                                    "common": "Estados Unidos",

                                    "official": "Estados Unidos da América"

                        },

                        "rus": {

                                    "common": "Соединённые Штаты Америки",

                                    "official": "Соединенные Штаты Америки"

                        },

                        "spa": {

                                    "common": "Estados Unidos",

                                    "official": "Estados Unidos de América"

                        }

            }

}

You can see above that this data is a typical MongoDB record with the auto-generated $oid. The Export-Import utility can read this MongoDB JSON format and load it into Oracle NoSQL Database. Let us look at how easy and with minimal steps this can be performed.

  • The Export/Import utility requires a JSON represented config file. You can customize your import operation by including one or many of the following options:

{

    "configFileVersion": <version>,

    "abortOnError": [false | true],

    "errorOutput": <error-dir>,

    "errorFileSizeMB": <error-file-chunk-size-mb>,

    "errorFileCount": <error-file-count>,

    "path": <dir-or-file>,               

    "namespace": <namespace>, 

    "tableName": <table-name>,                

    "ignoreFields": [<field1>, <field2>, ...],

    "renameFields": {

        <old-name>:<new-name>,

        ...

    }

    "charset": <charset-name>,

    "ddlSchemaFile": <file>,

    "continueOnDdlError": <false | true>,

    "streamConcurrency": <stream-parallelism>,

    "overwrite": <false | true>,

    “ttlRelativeDate”: <date-to-use in UTC>,

    “dateTimeToLong”: <true | false>

}

The most frequently used config options are

path – path to the JSON data file

Namespace – the namespace in Oracle NoSQL Database that you would like the imported table to be created.

IgnoreFields – the attributes in the JSON document that you would want to ignore

renameFields – the attributes in the JSON document that you would like to rename in the table in Oracle NoSQL Database.

ddlSchemaFile – point this to the table schema DDL script file

  • Run the import utility

java -jar kvtool.jar import -helper-hosts localhost:5000 -store kvstore -external -format MONGODB_JSON -config ./mongoimp/config/config.json –verbose

The parameters for the kvtool.jar are:

-import – to specify that the current operation is an Import.

-helper-hosts – the Oracle NoSQL Database connection details.

-store – the registered name of the kvstore

-external specifies that the data to import has been generated by a source other than Oracle NoSQL Database.

-format specifies the format to import.

-config – the location of the config file explained above.

With just 2 steps you would be able to migrate your MongoDB data to Oracle NoSQL Database.

Happy migration!

Join the discussion

Comments ( 1 )
  • valmir Monday, July 27, 2020
    These features are available to do it on NoSQLe (OCI version)?
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.