Here at Oracle, we have received questions regarding TDE Master Key rotation and Oracle Wallet password rotation in RAC environments. In this blog post, I will explain one of several possible ways to achieve this.
On an Oracle RAC cluster running a TDE encrypted database, rotate the master key and the wallet password. Repeat the rotation once or twice per year based on your corporate policy.
This is one of several possible "blueprints" for doing TDE master key and wallet password rotation on Oracle RAC environments. There are certainly other ways to do it depending on your RAC configuration, storage configuration, database versions, etc. The blueprint described below is broadly applicable given the RAC environments that many customers are running now.
Oracle end-user documentation makes additional recommendations for RAC and TDE. Note that alternatives described in the end-user documentation are useful for certain environments.
Let me first describe what a typical RAC+TDE environment would look like.
Oracle RAC system running on ASM, with no ACFS installed
The database is Oracle Database Enterprise Edition 10gR2, 11g or 11gR2
In the example that follows, we will be working with a database named rac-hr, which has a rac-hr1 instance running on node 1 and a rac-hr2 instance running on node 2
The database is encrypted only using TDE (tablespace or column), nothing else
TDE is using a standard symmetric key for the TDE master key, not a certificate (legacy)
For older database releases 11g and 10gR2, separate column encryption master key and tablespace encryption master key both must be rotated
-- On 10gR2, only column encryption is available, so only the column encryption master key can be rotated
-- On 11g, the command to rotate master key will execute successfully on the column encryption master key but will skip rotation for the tablespace encryption master key (note: master key rotation is not supported specifically for tablespace on 11g)
-- On 11gR2, there is a unified master encryption key that serves both column encryption and tablespace encryption. It will be rotated
Each RAC node has local sqlnet.ora files. There is one sqlnet.ora file for each encrypted database running on the node. The directory layout for storage of the sqlnet.ora files is the same on each node
-- Example: /u01/app/oracle/product/11.2/rac-hr/network/admin/sqlnet.ora
Inside of sqlnet.ora for a given database, the ENCRYPTION_WALLET_LOCATION setting points to a storage location for the wallet associated with that database. It is a unique and fully qualified location (path / subdirectory) on the node’s local file system. This setup avoids the danger of multiple databases attempting to share the same wallet file. The directory layout for storage of wallet files is the same on each RAC node
-- Example: ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /etc/ORACLE/WALLETS/rac-hr/)))
You must verify permissions on full directory paths and wallet files themselves to make sure each database can successfully access its wallet
-- Example: On Linux, create the wallet subdirectory as follows –
$> cd /etc
$> mkdir –pv ORACLE/WALLETS/rac-hr
$> chown –R oracle:oinstall ORACLE
$> chmod –R 700 ORACLE
RAC srvctl commands are used to startup the database. First, you start the database using srvctl start, and then you set the TNS_ADMIN environment variable using srvctl setenv. This informs the database where to look for its sqlnet.ora file across all of the nodes (i.e. recall that the directory layout is the same everywhere)
-- Example: $> srvctl setenv database –d rac-hr –t TNS_ADMIN=/u01/app/oracle/product/11.2/rac-hr/network/admin
Copies of both P12 and SSO are on each RAC node
The SSO used is a *local* auto-login wallet
TDE Wallets are used only for TDE (no storage of network encryption certificates or SEPS passwords)
An additional copy of the current P12 is stored and appropriately tagged/labeled within a separate backup archive
-- This can be any file archive, however for better security, it should be a separate archive location from backups of the TDE encrypted data. To tag/label the P12 backup, use description fields provided by your backup software or simply change the file name. For example, you could name the P12 file backup "<<description and timestamp>>.p12". If you ever need to restore the P12 file from archive, make sure to change the name back to "ewallet.p12"
ABOUT THE ROTATION PROCEDURE
In Oracle RAC environments, TDE operations such as wallet open/close are synchronized automatically across the cluster. However, when you create a new TDE master key and write it into the wallet or you rotate the wallet password, although these operations are executed successfully on the local RAC node (and corresponding changes are written to the local copy of the wallet), they do not fully synchronize across the cluster. The updated wallet must be copied manually from the RAC node where the rotation operations were performed to the other nodes.
These key and password rotation operations typically are infrequent and often part of larger planned maintenance activities, so the burden of performing additional copy steps is low. But remembering to do the file copy is critical. If the updated wallet is not copied over to other RAC nodes, then when they attempt to service queries on TDE encrypted data, they may fail with errors (cannot unwrap data encryption keys with new master key). Moreover, the next time other nodes attempt to open the wallet using a password (where the wallet has been set to a new password), again they will fail with errors.
Rotates both the TDE master key and the Oracle Wallet password
Rotates on a single database, but the procedure easily can be adapted/repeated to rotate multiple databases running on the cluster
Assumes that the cluster is using *local* auto-login wallet for benefits including unattended database startup and additional security
Procedure can be done with near-zero downtime for databases that are running on the cluster
Procedure can be done with near-zero chance of running queries returning errors during rotation
Note that the procedure involves temporarily bringing the cluster down to a single node, which means it should be performed during a time window when the cluster is lightly used (to not overwhelm the single node). Alternatively, you can perform the procedure during a planned/scheduled maintenance window when no queries are going to the cluster at all
DETAILED ROTATION STEPS
1. STARTING STATE: Encrypted database is running, with a live instance on each node
2. Backup the current P12 to archive
3. Identify one RAC node that will be the lead node
4. Use orapki wallet display -wallet to see master key list and validate the password
5. Bring all nodes down except for the lead node
Bringing down means to stop the database instance (the instance of the TDE encrypted database that requires rotation) running on the node
Example: $> srvctl stop instance -d rac-hr -i rac-hr2
6. On the lead node, rotate the TDE master key using sql command
This updates the running database and immediately writes a change into the P12 and SSO files
Example: sql> alter system set encryption key identified by "password"
7. Use orapki wallet display -wallet to see that a new master key has been added
8. Backup the P12 to archive
9. Close the wallet
This closes the updated P12, which is left open by the rotation command. Need to make sure the P12 is closed so that the updated SSO can open. After closing P12, when a query comes in, the updated SSO will open automatically
Example: sql> alter system set encryption wallet close identified by “password”
10. Run a simple query that touches TDE encrypted data
This makes sure that the updated SSO is in an open state (if it is not already open because a query has immediately come to the node that opened it)
It also serves as a checkpoint to make sure we are successful up to this point and the node still can query TDE encrypted data fine
11. On the lead node, rotate the wallet password using orapki utility
This updates the running database and immediately writes a change into the P12 and SSO files
Example: $> orapki wallet change_pwd -wallet /etc/ORACLE/WALLETS/rac-hr
12. Use orapki wallet display -wallet to validate the new password
13. Backup the P12 to archive
14. Close the wallet
This closes the SSO, which is left open by the previous query on TDE encrypted data. Need to make sure the older wallet is flushed from memory and the newer SSO can open. After closing older SSO, when a query comes in, the newer SSO will be opened automatically
Example: sql> alter system set encryption wallet close
Do not need the clause identified by “password” because it was an SSO that was open and no password required
15. Run a simple query that touches TDE encrypted data
Ensures that updated SSO can be opened, also serves as checkpoint
16. On the other nodes, delete their local copies of P12 and SSO
17. Copy the updated P12 from the lead node to other nodes
18. Go to other nodes one-at-a-time
Regenerate the *local* auto-login wallet (SSO)
Example: orapki wallet create -wallet /etc/ORACLE/WALLETS/rac-hr -auto_login_local
Verify permissions for full path and wallet files themselves. Make sure wallet files can be accessed by the database
19. Bring other nodes back up
Example: $> srvctl start instance -d rac-hr -i rac-hr2
20. After all nodes are back up, from the lead node, close wallet one last time and run a simple query on TDE encrypted data
Ensures that updated SSO can be opened by nodes, also serves as checkpoint
21. ENDING STATE: Encrypted database is running, with a live instance on each node. Master key and wallet password have been rotated
Hope you find this blog post helpful. Please feel free to leave your comments and suggestions.