X

Maximum Availability Architecture – Oracle’s industry-leading set of database high availability capabilities

Migrating RMAN backups from S3 to OCI (OSB Cloud Module for AWS S3 to DB Cloud Backup Module for OCI)

Marco Calmasini
Sr. Principal Product Manager

 

In part 1 of this blog series, we will outline the procedure for moving existing Oracle database backups residing on AWS S3 to OCI.

 

Why move Oracle Database Backups to OCI?

The OSB Cloud Module for AWS S3 does not support Glacier (archive storage), your backups can only be stored in S3 with the associated costs, which can become very high, especially over the long term.

The Database Backup Service in OCI does support Object Lifecycle Policies. Backups can be stored in standard Object Storage for a defined period of time for quick restore, and then automatically tiered to Archive Storage for long-term, low-cost retention. RMAN automatically manages recalling and restoring backups from Archive Storage as needed.

The Backup  Service in OCI does not require per-stream licenses; they are required with the OSB Cloud Module for AWS S3. The Database Backup Cloud Module for OCI is included as part of the service and can leverave as many streams as necessary to achieve your performance goals, across any number of databases.

Because the OSB Cloud Module for AWS S3 and the DB Backup Cloud Module for OCI use the same format when storing backup data in cloud buckets, migrating backups from S3 to OCI is a straightforward process as shown below.

How RMAN backup sets map to OCI objects

RMAN uploads backup sets to cloud buckets as a number of backup pieces, and each backup piece maps to one or more objects in the cloud bucket – these associations are recorded in the metadata.xml manifest file.  The number of objects depends on the size of the backup piece. By default, a backup piece up to 10GB in size is stored in OCI as a single object, larger backup pieces will use multiple objects.

The backup piece name is controlled by the FORMAT parameter in RMAN. The recommendation is to include at least ‘%d_%U’ to make sure each object name is unique.

%d -> DB_NAME

%U -> system generated unique identifier

For example: ORCL_ctua720h_1_1 is a backup piece handle created using the ‘%d_%U’ format.

The objects in the cloud bucket for such a backup piece will look like this.

sbt_catalog/ORCL_ctua720h_1_1/metadata.xml  (metadata manifest file)

file_chunk/<DBID>/<DBNAME>/backuppiece/<DATE>/ORCL_ctua720h_1_1/<INCARNATION>/<CHUNK#>

file_chunk/<DBID>/<DBNAME>/backuppiece/<DATE>/ORCL_ctua720h_1_1/<INCARNATION>/metadata.xml (copy of metadata.xml file)

1. Copying data to OCI

All the objects pertaining to a specific backup piece must be copied for RMAN to be able to restore it. The chunks where the actual backup data are and the two metadata.xml files.

To copy objects from the AWS S3 bucket to OCI you can use rclone available via ­­http://rclone.org.

It can be installed and used on any system that has access to both the AWS S3 bucket and the OCI Object Storage bucket.

First of all, you need to obtain:

  • S3 ACCESS KEY and SECRET ACCESS KEY for the AWS S3 user
    • If you don’t have them, go to the AWS console ->IAM->Users->Security Credentials and click on Create Keys
  • S3 REGION where the bucket is located
  • S3 BUCKET name for the bucket where the backups are located
  • OCI STORAGE NAMESPACE
    • Go to Profile on the top right corner of the OCI console and select Tenancy:<your tenancy name> in the Tenacy Information section.  Under Object Storage Settings, you will see the Object Storage Namespace. The namespace value can be either the tenacy name or a randomly generated string. The name cannot be changed.
  • OCI Amazon S3 Compatibility API Designated Compartment
    • In the same Object Storage Settings of the Tenancy Information section you will also find the compartment where your OCI destination bucket must be located. If you want to change the designated Compartment click on Edit Object Storage Settings at the top of the page.
  • OCI CUSTOMER ACCESS KEY and SECRET KEY
    • If you don’t have them, go to Profile on the top right corner of the OCI console and select User Settings, then on the bottom left of the screen, under Resources click on Customer Secret Keys and then click on Generate Secret Key. Make a note of the Secret Key as it’s displayed only once.
  • OCI REGION where you want your target bucket to be located

Before starting the copy with rclone, we have to set some environment variables to define the source and target locations based on the information above:

# export RCLONE_CONFIG_S3_TYPE=s3

# export RCLONE_CONFIG_S3_ACCESS_KEY_ID=AKIRGGSJRV23S5AG4N

# export RCLONE_CONFIG_S3_SECRET_ACCESS_KEY=TLJkltRDASlSlhVRPsRuJse2FtWLnFD5

# export RCLONE_CONFIG_S3_REGION=<region>   for example: us-east-1

# export SOURCE=s3:<bucket name>   

 

# export RCLONE_CONFIG_OCI_TYPE=s3     rclone uses S3 compatibility APIs to access OCI Object Storge

# export RCLONE_CONFIG_OCI_ACCESS_KEY_ID=b8d65742ca7385eac091f1c0e86376d1e30eb4

# export RCLONE_CONFIG_OCI_SECRET_ACCESS_KEY=26TtH1CSSFgddsEPwDoBqPCsLVrapmerolAsDg=

# export RCLONE_CONFIG_OCI_REGION=<region>      for example: us-ashburn-1

# export RCLONE_CONFIG_OCI_ENDPOINT=https://<namespace>.compat.objectstorage.<region>.oraclecloud.com

Once everything is set up you can start the rclone copy process, all the content of the AWS S3 bucket will be copied to the OCI target. If the bucket does not exist it is created.

 

# rclone --verbose --cache-workers 64 --transfers 64 --retries 32 copy $SOURCE oci:<bucketname>


2. Installing the DB Backup Cloud Module

Once the copy is complete, you will find all the backup objects in the bucket you designated.  In order to be able to restore from OCI you have to install the DB Backup Cloud Module and configure it to point to the new OCI bucket.

The DB Backup Cloud Module installer is available at https://www.oracle.com/database/technologies/oracle-cloud-backup-downloads.html

Follow the documentation available at https://docs.oracle.com/en/cloud/paas/db-backup-cloud/csdbb/installing-oracle-database-cloud-backup-module-oci.html for details on running the installer,

You also have the option to set a lifecycle policy on the bucket in order to move the backups to archive storage after a period of time that can be between 0 days (data will be archived within 24 hours) and number of years. This can be done using –enableArchiving, –archiveAfterBackup, and –retainAfterRestore parameters.

Note: Do not create the policy manually using the OCI console. Instead, use the appropriate archiving options mentioned above for the DB Backup Cloud Module installer so that the appropriate exclusion for .xml files is also added.

The DB Backup Cloud Module for OCI can co-exist with the OSB Cloud Module for AWS S3 if you want to be able to access both buckets for a period of time.

RMAN can now be configured to use the newly installed module, referencing the new libopc.so SBT library and using the new configuration file created by the installer.

The backups in the OCI bucket are now accessible and an RMAN restore validate database command can be executed to verify the process was successful.

3. Restoring from archive storage

Since data residing in the archive storage tier cannot be immediately accessed by RMAN clients, it must first be restored (or recalled, in RMAN terminology) to standard object storage. This process can take 1 hour or more depending on the volume of data. The objects go from an "Archived" state to "Restoring" and then "Restored". When an object is "Restored" it means it's in standard object storage and RMAN can access it.

RMAN can initiate this recall process through the ‘restore database preview recall’ command.

Allternatively, the RMAN restore database preview command can be used to just determine if the backup pieces required for a database restore operation are immediately available in standard object storage, or if recalling backups from archive storage tier is required. In this example, an RMAN restore database preview command output shows that a certain backup piece is archived and needs to be recalled.

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------

179     Full    256.00K    SBT_TAPE    00:00:02     14-SEP-19

        BP Key: 179   Status: AVAILABLE  Compressed: NO  Tag: TAG20190914T100406

        Handle: 89ubntom_1_1   Media: objectstorage.us-ashburn-..ecloud.com/n/oradbclouducm/tdemoaug22

  List of Datafiles in backup set 179

  File LV Type Ckp SCN    Ckp Time  Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- --------- ----------- ------ ----

  21      Full 4715399    14-SEP-19              NO    /ade/b/3380669573/oracle/dbs/tbs_25.f

using channel ORA_SBT_TAPE_1

using channel ORA_DISK_1

 

archived logs generated after SCN 4715399 not found in repository

recovery will be done up to SCN 4715399

Media recovery start SCN is 4715399

Recovery must be done beyond SCN 4715399 to clear datafile fuzziness

 

List of remote backup files

============================

        Handle: 89ubntom_1_1   Media: objectstorage.us-ashburn-..ecloud.com/n/oradbclouducm/tdemoaug22

validation succeeded for backup piece

Finished restore at 14-SEP-19

 

If one or more files are listed as “remote” you can either issue a RMAN restore database preview recall command or just RMAN restore database  command. While the restore database preview recall command will initiate the recall from archive storage and exit, the restore database command will also initiate the recall from archive process and then wait for it to be completed before proceeding with the actual database restore.

The following example shows the output of a restore database preview recall  command:

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------

179     Full    256.00K    SBT_TAPE    00:00:02     14-SEP-19

        BP Key: 179   Status: AVAILABLE  Compressed: NO  Tag: TAG20190914T100406

        Handle: 89ubntom_1_1   Media: objectstorage.us-ashburn-..ecloud.com/n/oradbclouducm/tdemoaug22

  List of Datafiles in backup set 179

  File LV Type Ckp SCN    Ckp Time  Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- --------- ----------- ------ ----

  21      Full 4715399    14-SEP-19              NO    /ade/b/3380669573/oracle/dbs/tbs_25.f

using channel ORA_SBT_TAPE_1

using channel ORA_DISK_1

 

archived logs generated after SCN 4715399 not found in repository

recovery will be done up to SCN 4715399

Media recovery start SCN is 4715399

Recovery must be done beyond SCN 4715399 to clear datafile fuzziness

 

Initiated recall for the following list of remote backup files

==========================================================

        Handle: 89ubntom_1_1   Media: objectstorage.us-ashburn ..ecloud.com/n/oradbclouducm/tdemoaug22

validation succeeded for backup piece

Finished restore at 14-SEP-19

In this blog we showed how to move you backups taken using the Oracle Secure Backup Cloud Module for AWS S3 from an S3 bucket to an OCI Object Storage bucket in order to be restored using the Oracle Database Backup Cloud Module. These backups could be coming from a database running in an AWS EC2 compute instance or from and on-premises database and can be restored back to on-premises or to an OCI instance (Compute, DBaaS or ExaCS). For restoring into DBaaS or ExaCS some limitations apply on versions, DB architecture etc. Check the OCI documentation for more information https://docs.cloud.oracle.com/en-us/iaas/Content/Database/Tasks/migrating.htm

In the next blog we will cover how to migrate backups from an AWS RDS Oracle instance to OCI.

 

 

 

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.