Oracle RAC: Rotating the TDE Master Key and Oracle Wallet Password with a Wallet Copy on Each Node
By Saikat Saha-Oracle on Apr 17, 2014
Here at Oracle, we have received questions regarding TDE Master Key rotation and Oracle Wallet password rotation in RAC environments. In this blog post, I will explain one of several possible ways to achieve this.
On an Oracle RAC cluster running a TDE encrypted database, rotate the master key and the wallet password. Repeat the rotation once or twice per year based on your corporate policy.
This is one of several possible "blueprints" for doing TDE master key and wallet password rotation on Oracle RAC environments. There are certainly other ways to do it depending on your RAC configuration, storage configuration, database versions, etc. The blueprint described below is broadly applicable given the RAC environments that many customers are running now.
Oracle end-user documentation makes additional recommendations for RAC and TDE. Note that alternatives described in the end-user documentation are useful for certain environments.
TARGET ENVIRONMENTLet me first describe what a typical RAC+TDE environment would look like.
-- On 10gR2, only column encryption is available, so only the column encryption master key can be rotated
-- On 11g, the command to rotate master key will execute successfully on the column encryption master key but will skip rotation for the tablespace encryption master key (note: master key rotation is not supported specifically for tablespace on 11g) -- On 11gR2, there is a unified master encryption key that serves both column encryption and tablespace encryption. It will be rotated
-- Example: /u01/app/oracle/product/11.2/rac-hr/network/admin/sqlnet.ora
-- Example: ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /etc/ORACLE/WALLETS/rac-hr/)))
-- Example: On Linux, create the wallet subdirectory as follows –
$> cd /etc
$> mkdir –pv ORACLE/WALLETS/rac-hr
$> chown –R oracle:oinstall ORACLE
$> chmod –R 700 ORACLE
-- Example: $> srvctl setenv database –d rac-hr –t TNS_ADMIN=/u01/app/oracle/product/11.2/rac-hr/network/admin
-- This can be any file archive, however for better security, it should be a separate archive location from backups of the TDE encrypted data. To tag/label the P12 backup, use description fields provided by your backup software or simply change the file name. For example, you could name the P12 file backup "<<description and timestamp>>.p12". If you ever need to restore the P12 file from archive, make sure to change the name back to "ewallet.p12"
ABOUT THE ROTATION PROCEDURE
In Oracle RAC environments, TDE operations such as wallet open/close are synchronized automatically across the cluster. However, when you create a new TDE master key and write it into the wallet or you rotate the wallet password, although these operations are executed successfully on the local RAC node (and corresponding changes are written to the local copy of the wallet), they do not fully synchronize across the cluster. The updated wallet must be copied manually from the RAC node where the rotation operations were performed to the other nodes.
These key and password rotation operations typically are infrequent and often part of larger planned maintenance activities, so the burden of performing additional copy steps is low. But remembering to do the file copy is critical. If the updated wallet is not copied over to other RAC nodes, then when they attempt to service queries on TDE encrypted data, they may fail with errors (cannot unwrap data encryption keys with new master key). Moreover, the next time other nodes attempt to open the wallet using a password (where the wallet has been set to a new password), again they will fail with errors.
DETAILED ROTATION STEPS
1. STARTING STATE: Encrypted database is running, with a live instance on each node
2. Backup the current P12 to archive
3. Identify one RAC node that will be the lead node
4. Use orapki wallet display -wallet to see master key list and validate the password
5. Bring all nodes down except for the lead node
6. On the lead node, rotate the TDE master key using sql command
7. Use orapki wallet display -wallet to see that a new master key has been added
8. Backup the P12 to archive
9. Close the wallet
10. Run a simple query that touches TDE encrypted data
11. On the lead node, rotate the wallet password using orapki utility
12. Use orapki wallet display -wallet to validate the new password
13. Backup the P12 to archive
14. Close the wallet
15. Run a simple query that touches TDE encrypted data
16. On the other nodes, delete their local copies of P12 and SSO
17. Copy the updated P12 from the lead node to other nodes
18. Go to other nodes one-at-a-time
19. Bring other nodes back up
20. After all nodes are back up, from the lead node, close wallet one last time and run a simple query on TDE encrypted data
21. ENDING STATE: Encrypted database is running, with a live instance on each node. Master key and wallet password have been rotated
Hope you find this blog post helpful. Please feel free to leave your comments and suggestions.