In part 1 of this blog series, we will outline the procedure for moving existing Oracle database backups residing on AWS S3 to OCI.
Why move Oracle Database Backups to OCI?
The OSB Cloud Module for AWS S3 does not support Glacier (archive storage), your backups can only be stored in S3 with the associated costs, which can become very high, especially over the long term.
The Database Backup Service in OCI does support Object Lifecycle Policies. Backups can be stored in standard Object Storage for a defined period of time for quick restore, and then automatically tiered to Archive Storage for long-term, low-cost retention. RMAN automatically manages recalling and restoring backups from Archive Storage as needed.
The Backup Service in OCI does not require per-stream licenses; they are required with the OSB Cloud Module for AWS S3. The Database Backup Cloud Module for OCI is included as part of the service and can leverave as many streams as necessary to achieve your performance goals, across any number of databases.
Because the OSB Cloud Module for AWS S3 and the DB Backup Cloud Module for OCI use the same format when storing backup data in cloud buckets, migrating backups from S3 to OCI is a straightforward process as shown below.
How RMAN backup sets map to OCI objects
RMAN uploads backup sets to cloud buckets as a number of backup pieces, and each backup piece maps to one or more objects in the cloud bucket – these associations are recorded in the metadata.xml manifest file. The number of objects depends on the size of the backup piece. By default, a backup piece up to 10GB in size is stored in OCI as a single object, larger backup pieces will use multiple objects.
The backup piece name is controlled by the FORMAT parameter in RMAN. The recommendation is to include at least ‘%d_%U’ to make sure each object name is unique.
%d -> DB_NAME
%U -> system generated unique identifier
For example: ORCL_ctua720h_1_1 is a backup piece handle created using the ‘%d_%U’ format.
The objects in the cloud bucket for such a backup piece will look like this.
sbt_catalog/ORCL_ctua720h_1_1/metadata.xml (metadata manifest file)
file_chunk/<DBID>/<DBNAME>/backuppiece/<DATE>/ORCL_ctua720h_1_1/<INCARNATION>/metadata.xml (copy of metadata.xml file)
1. Copying data to OCI
All the objects pertaining to a specific backup piece must be copied for RMAN to be able to restore it. The chunks where the actual backup data are and the two metadata.xml files.
To copy objects from the AWS S3 bucket to OCI you can use rclone available via http://rclone.org.
It can be installed and used on any system that has access to both the AWS S3 bucket and the OCI Object Storage bucket.
First of all, you need to obtain:
Before starting the copy with rclone, we have to set some environment variables to define the source and target locations based on the information above:
# export RCLONE_CONFIG_S3_TYPE=s3
# export RCLONE_CONFIG_S3_ACCESS_KEY_ID=AKIRGGSJRV23S5AG4N
# export RCLONE_CONFIG_S3_SECRET_ACCESS_KEY=TLJkltRDASlSlhVRPsRuJse2FtWLnFD5
# export RCLONE_CONFIG_S3_REGION=<region> for example: us-east-1
# export SOURCE=s3:<bucket name>
# export RCLONE_CONFIG_OCI_TYPE=s3 rclone uses S3 compatibility APIs to access OCI Object Storge
# export RCLONE_CONFIG_OCI_ACCESS_KEY_ID=b8d65742ca7385eac091f1c0e86376d1e30eb4
# export RCLONE_CONFIG_OCI_SECRET_ACCESS_KEY=26TtH1CSSFgddsEPwDoBqPCsLVrapmerolAsDg=
# export RCLONE_CONFIG_OCI_REGION=<region> for example: us-ashburn-1
# export RCLONE_CONFIG_OCI_ENDPOINT=https://<namespace>.compat.objectstorage.<region>.oraclecloud.com
Once everything is set up you can start the rclone copy process, all the content of the AWS S3 bucket will be copied to the OCI target. If the bucket does not exist it is created.
# rclone --verbose --cache-workers 64 --transfers 64 --retries 32 copy $SOURCE oci:<bucketname>
2. Installing the DB Backup Cloud Module
Once the copy is complete, you will find all the backup objects in the bucket you designated. In order to be able to restore from OCI you have to install the DB Backup Cloud Module and configure it to point to the new OCI bucket.
The DB Backup Cloud Module installer is available at https://www.oracle.com/database/technologies/oracle-cloud-backup-downloads.html
Follow the documentation available at https://docs.oracle.com/en/cloud/paas/db-backup-cloud/csdbb/installing-oracle-database-cloud-backup-module-oci.html for details on running the installer,
You also have the option to set a lifecycle policy on the bucket in order to move the backups to archive storage after a period of time that can be between 0 days (data will be archived within 24 hours) and number of years. This can be done using –enableArchiving, –archiveAfterBackup, and –retainAfterRestore parameters.
Note: Do not create the policy manually using the OCI console. Instead, use the appropriate archiving options mentioned above for the DB Backup Cloud Module installer so that the appropriate exclusion for .xml files is also added.
The DB Backup Cloud Module for OCI can co-exist with the OSB Cloud Module for AWS S3 if you want to be able to access both buckets for a period of time.
RMAN can now be configured to use the newly installed module, referencing the new libopc.so SBT library and using the new configuration file created by the installer.
The backups in the OCI bucket are now accessible and an RMAN restore validate database command can be executed to verify the process was successful.
3. Restoring from archive storage
Since data residing in the archive storage tier cannot be immediately accessed by RMAN clients, it must first be restored (or recalled, in RMAN terminology) to standard object storage. This process can take 1 hour or more depending on the volume of data. The objects go from an "Archived" state to "Restoring" and then "Restored". When an object is "Restored" it means it's in standard object storage and RMAN can access it.
RMAN can initiate this recall process through the ‘restore database preview recall’ command.
Allternatively, the RMAN restore database preview command can be used to just determine if the backup pieces required for a database restore operation are immediately available in standard object storage, or if recalling backups from archive storage tier is required. In this example, an RMAN restore database preview command output shows that a certain backup piece is archived and needs to be recalled.
If one or more files are listed as “remote” you can either issue a RMAN restore database preview recall command or just RMAN restore database command. While the restore database preview recall command will initiate the recall from archive storage and exit, the restore database command will also initiate the recall from archive process and then wait for it to be completed before proceeding with the actual database restore.
The following example shows the output of a restore database preview recall command:
In this blog we showed how to move you backups taken using the Oracle Secure Backup Cloud Module for AWS S3 from an S3 bucket to an OCI Object Storage bucket in order to be restored using the Oracle Database Backup Cloud Module. These backups could be coming from a database running in an AWS EC2 compute instance or from and on-premises database and can be restored back to on-premises or to an OCI instance (Compute, DBaaS or ExaCS). For restoring into DBaaS or ExaCS some limitations apply on versions, DB architecture etc. Check the OCI documentation for more information https://docs.cloud.oracle.com/en-us/iaas/Content/Database/Tasks/migrating.htm
In the next blog we will cover how to migrate backups from an AWS RDS Oracle instance to OCI.