X

Information, tips, tricks and sample code for Big Data Warehousing in an autonomous, cloud-driven world

How To: Download all files from an export to object store job in Autonomous Database using cURL

Nilay Panchal
Senior Product Manager

A good portion of Autonomous Database (ADB) users export and store their data in Oracle Cloud Object Storage. Last year, I had written this guide to help users export database dump files, using the Data Pump tool, to Oracle's object-store. Once completed, an export, depending on the export filesize parameters, may have created several dump file parts. You may then download your dump file parts using a tool with Swift API support such as the popular cURL tool; most newer operation systems have cURL preinstalled.

Recently, we have had users, that use cURL to download their files, reach out to ask how they may download all their exported dump file parts at once. cURL lacks support for wildcard or substitution characters in its URL, so it isn't a simple one-line command. Since my job is to make your life easier, below I provide a sample, easy-to-invoke bash script to download the multiple dump file parts of an export job from the object store, using cURL.

 

Note: This example below is run on a Mac and works similarly on Linux. If you are on Windows, you may need to install PowerShell, Cygwin or similar to run shell scripts.

 

Step 1: Download the bash script

Once you have run your export job and the dump file parts are in the object store, download the shell script “get_objects.sh” by clicking this link and unzipping it. You may open this script in a text editor and read the detailed notes and description of what this script does and the parameters it accepts.

 

Step 2: Run the script with your object store URL and credential parameters.

Once you have downloaded the get_objects.sh script, open the Terminal and navigate to the folder where you downloaded the script above. Run the script with bash command below, replacing in:

  1. The SWIFT object store URL that you used in your export job following the "-f" file parameter. Notice, we also include using the same substitution character “%U” in that URL to download all the dump files from an export job. In this example, the dump file part names begin with “exp” and end in “.dmp”, the dump file extension.
  2. Your user credentials following the “-a” authorization parameter to access the object store, which is your username and SWIFT Auth token, as used in your export job.
    If you do not have this information, you may create a new credential by following Step 7 in this ADB workshop.

 

bash get_objects.sh -f 'https://swiftobjectstorage.<region identifier>.oraclecloud.com/v1/<object store namespace>/<bucketname>/exp%U.dmp' -a '<username>':'<SWIFT auth token>' –verbose

 

 

Once executed, the script will begin downloading all the dump files that match the name pattern in the URL provided! Follow the notes in the bash script if you need to use additional parameters in this bash invocation, for things like specifying your proxy or downloading to a specific folder.

I hope this time-saving example helps many of you easily download your exported dump files from the object store in one easy go!

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.