Wednesday Dec 21, 2016

APEX is in CDB$ROOT again - Journey to the Cloud VII

Journey to the Oracle CloudWell ... it's been a while ... but I would like to continue my journey to the cloud ...

What happened so far on my Journey to the Cloud?

Oracle Database 12.2.0.1 in the Cloud

Since November 2016 Oracle Database 12.2.0.1 is available in the Oracle DBaaS Cloud. And I received this question in my inbox yesterday:

I have a customer who wants to migrate Apex 4.2 applications to DBCS and ORDS.war to Weblogic on Compute.

I recently went through your blog on removing Apex from CDB Root. The customer is planning to do something similar but had questions on repercussions of doing so in Oracle Public Cloud.

What are the factors that need to be considered. Also how would DBCS patching work in this scenario.

.

Is APEX is in CDB$ROOT again

I haven't checked this yet as we've had a very productive conversation with the APEX folks a while back. And I'm 100% sure that the APEX group wasn't involved in this decision as they recommend clearly in the doc to NOT HAVE APEX installed in the CDB$ROOT ;-)
Ok, I did connect to our Cloud environment and ... voila ...

COLUMN VERSION FORMAT A15
COLUMN NAME FORMAT A12
COLUMN COMP_ID FORMAT A10

SQL> select r.comp_id, r.version, c.name from cdb_registry r, v$containers c where r.con_id=c.con_id and r.comp_id='APEX' order by 3

COMP_ID    VERSION         NAME
---------- --------------- ------------
APEX       5.0.4.00.12     CDB$ROOT
APEX       5.0.4.00.12     ROYPDB1

Ouch ...

The presence of APEX in the CDB$ROOT may have to do with the Oracle DBaaS Monitor Console. This is just an assumption but when I removed the APEX in my old 12.1.0.2 cloud deployment I had to clean-up the DBaaS Monitor as well.

See:

for my experience a few months back.
.

Solution?

Well, having APEX in the CDB$ROOT is still a brilliant [IRONY!] idea. As soon as you start unplug/plug operations with APEX in the PDB only or with a different APEX version you are asking for trouble.

See this blog post for the potential pitfalls:

Which options does the customer have assuming that his APEX 4.2 application is in a non-CDB?

  • Upgrade APEX locally to 5.0.4.00.12 before migrating the database to the DBaaS cloud
    This would be the easiest workaround if it wouldn't involve an application software upgrade. Through the Oracle glasses this looks simple - but from a customer's and application developer's standpoint this isn't trivial as most likely it will involve application testing
    .
  • Export the APEX application and import it
    I haven't done this by myself but first of all with APEX 4.2 (or below) you must take care to move the image files as well - and you'll have to move data as well. And, of course, you won't end up in APEX 4.2 but in APEX 5.0 so the above mentioned application upgrade will hit you as well. I don't see any benefit over solution 1.
    .
  • Remove APEX from the 12.2.0.1 DBaaS Cloud deployment's CDB$ROOT
    This is - in my humble opinion - the only viable solution here if the customer can't upgrade APEX in the current environment for whatever reason. But this will most likely remove the DBaaS Monitor as well. I can live without it but I know that it offers a lot of good stuff especially when dealing with encrypted tablespaces which is otherwise hard to handle on the command line. The good part of this solution is the freedom and flexibility you'll get once APEX is removed from the CDB$ROOT for unplug/plug operations in the future.
    .

Finally, regarding patching:
I don't see any issues. And the same for a future upgrade as we decouple APEX upgrades from the database upgrade with Oracle Database 12.2.0.
.

--Mike




Monday Jun 27, 2016

Full Transportable Export/Import - Migrating an 11.2.0.4 database to Oracle Database 12c- into the Oracle Cloud

Full Transportable Export/Import - one of the coolest features in Oracle Database 12c 

We blogged about Full Transportable Export/Import a while back. It is - no doubt - one of the coolest features in Oracle Database 12c. And it is part of our Hands-On Lab exercise (Hands On Lab - Upgrade, Migrate, Consolidate to Oracle Database 12c) as well.

It utilizes the technique of Transportable Tablesspaces - cross-platform, cross- Endianness, cross-version - but lets Oracle Data Pump do all the "dirty" work of rebuilding everything kept in your SYSTEM and SYSAUX tablespace including views, synonyms, public objects, grants, sequences etc etc etc.

You can even combine it with RMAN Incremental Backups - of course cross-platform and cross-Endianness as well - in order to decrease the downtime for large databases.
Please find our presentation about the combination of these feature here in the Slides Download Center to the right:

In addition: You can use this Oracle Database 12c feature with your source database Oracle 11.2.0.3 or 11.2.0.4. Just the destination database you are upgrading or migrating to must be at least an Oracle 12.1.0.1 database (or newer of course). 

See the feature in action

Roy recorded a 20 minute video demonstrating how to:

  • Migrate an Oracle Database 11.2.0.4 database into an Multitenant's pluggable database using
  • Full Transportable Export/Import
  • into the Oracle DBaaS Cloud
    .

And of course the feature works also without migrating into a Container database environment - and without migrating to the Oracle DBaaS Cloud. 

If you need to migrate a database from Oracle 11.2.0.3 or newer to Oracle Database 12c Full Transportable Export/Import can become your best friend.

--Mike

Monday Apr 18, 2016

Patching does not work - Journey to the Cloud VI

DBaaS Oracle Cloud

What happened so far on my Journey to the Cloud?


Patching in the Cloud

I would like to patch my Oracle DBaaS Cloud today. It was so simple a few weeks ago. But I didn't patch it yet up to the January 2016 PSU (12.1.0.2.160119PSU) - shame on me :-(

Please don't be scared by the weird mix of German and English - I filed a bug for this months ago ... and maybe one magic sunny day I can switch the language back to pure English ...

Anyhow, I choose my VM and it highlighted to me an available patch to apply, the January 2016 PSU:


.

Let's do the PreCheck 

I chose PRECHECK first from the drop-down hamburger menu to the right and received the following message: 

But where are the logs? There is no link, no hint, nothing.
.

Let's check the README 

At this point I decided to check the README to find out whether I missed something important. So I clicked on README and received this meaningful message:

Potentially not a fault of the Cloud folks as I realized by myself that the MOS facility to link patches directly is broken since weeks. A manual interaction in MOS to locate the README is necessary - but it gave me no indication regarding the above failed precheck.
.

Force to apply the PSU 

Next step: FORCE to apply the patch (2nd option in the hamburger menu). But again the result is not positive showing me again a weird mix of Denglish (Deutsch and English).

.

Result?

Well, that is somewhat unexpected. I checked with some people who are way more familiar than I with our DBaaS Cloud after consulting our documentation and googling around - and learned that this functionality is broken since several weeks,

No further comment necessary.
.

--Mike

Thursday Jan 28, 2016

TDE is wonderful - Journey to the Cloud V

DBaaS Oracle Cloud

 What happened so far on my Journey to the Cloud?

Today's journey:
Learn about TDE (Transparent Data Encryption) and other secrets

What I really really love about my job: Every day I learn something new.

But sometimes learning can be frustrating at the beginning. And so it was for Roy and myself in the past days when we explored the use of TDE (Transparent Data Encryption) in our DBaaS Cloud environments. But many thanks to Brian Spendolini for his continuous 24x7 support :-)

Never heard of Transparent Data Encryption before? Then please read on here. It's usually part of ASO (Advanced Security Option) but it is included in the cloud offering. 

But first of all before taking care on TDE and PDBs I tried to deploy a new DBaaS VM ...
.

PDB names can't contain underscores?

Well, one learning experience is that initially you can't create a PDB in the DBaaS environment with an underscore in the name. I wanted to name my PDB in my new env simply "TDE_PDB1" (8 characters, all should be fine) - but received this nice message:

Don't worry if you don't speak German - it basically says that it can't be longer than 8 characters (ok, I knew that), must begin with a character (mine does of course) and can only contain characters and number (eh?? no underscores???). Hm ...?!?

Ok, I'll name mine "TDEPDB1".

Of course outside this page you can create PDBs containing an underscore:

SQL> create pluggable database PDB_MIKE admin user mike identified by mike
  2  file_name_convert=('/oradata/CDB2/pdbseed', '/oradata/CDB2/pdb_mike');

Pluggable database created
.

That's what happens when application logic tries to superseed database logic.
.

(Almost) undocumented parameter: encrypt_new_tablespace

Thanks to Robert Pastijn for telling me about this hidden secret. A new parameter which is not in the regular database deployment but only in the cloud.

encrypt_new_tablespaces

First check in MOS:

Interesting.

So let's check with Google.
And here it is: 7 hits, for example:

Controlling Default Tablespace Encryption

The ENCRYPT_NEW_TABLESPACES initialization parameter controls default encryption of new tablespaces. In Database as a Service databases, this parameter is set to CLOUD_ONLY.

Value
Description

ALWAYS

Any tablespace created will be transparently encrypted with the AES128 algorithm unless a different algorithm is specified on the ENCRYPTION clause.

CLOUD_ONLY

Tablespaces created in a Database Cloud Service database will be transparently encrypted with the AES128 algorithm unless a different algorithm is specified on the ENCRYPTION clause. For non-Database Cloud Service databases, tablespaces will only be encrypted if the ENCRYPTION clause is specified. This is the default value.

DDL

Tablespaces are not transparently encrypted and are only encrypted if the ENCRYPTION clause is specified.

What I found really scary is the fact that I couldn't find it in my spfile/pfile. You can alter it with an "alter system" command but you can't remove it.

The idea behind this is great as tablespaces should be encrypted, especially when they reside in a cloud environment. TDE is a very useful feature. And this mechanism exists regardless of your edition, whether you have Standard Edition or Enterprise Edition in any sort of flavor in the DBaaS Cloud.

A new tablespace will be encrypted by default:

SQL> CREATE TABLESPACE TS_MIKE DATAFILE 'ts_mike01.dbf' SIZE 10M;

Then check:

SQL> select TABLESPACE_NAME, ENCRYPTED from DBA_TABLESPACES;

But we'll see later if this adds some constraints to our efforts to migrate a database for testing purposes into the DBaaS cloud environment.
.

Is there anything encrypted yet?

Quick check after setting:

SQL> alter system set exclude_seed_cdb_view=FALSE scope=both;

I tried to find out if any of the tablespaces are encrypted.

SQL> select tablespace_name, encrypted, con_id from CDB_TABLESPACES order by 3;

TABLESPACE_NAME                ENC     CON_ID
------------------------------ --- ----------
SYSTEM                         NO           1
USERS                          NO           1
SYSAUX                         NO           1
UNDOTBS1                       NO           1
TEMP                           NO           1

SYSTEM                         NO           2
USERS                          NO           2
TEMP                           NO           2
SYSAUX                         NO           2

EXAMPLE                        NO           3
USERS                          NO           3
TEMP                           NO           3
SYSAUX                         NO           3
APEX_1701140435539813          NO           3
SYSTEM                         NO           3

15 rows selected.

Looks good.  Nothing encrypted yet.
.

How does the new parameter ENCRYPT_NEW_TABLESPACES effect operation?

Ok, lets try.

SQL> show parameter ENCRYPT_NEW_TABLESPACES

NAME                                 TYPE        VALUE
------------------------------------ ----------- ---------------
encrypt_new_tablespaces              string      CLOUD_ONLY

And further down the road ...

SQL> alter session set container=pdb1;

SQL> create tablespace MIKE_PLAYS_WITH_TDE datafile '/u02/app/oracle/oradata/MIKEDB/PDB1/mike_plays_with_tde.dbf' size 10M;

Tablespace created.

SQL> select tablespace_name, encrypted, con_id from CDB_TABLESPACES order by 3;

TABLESPACE_NAME                ENC     CON_ID
------------------------------ --- ----------
SYSTEM                         NO           3
SYSAUX                         NO           3
TEMP                           NO           3
USERS                          NO           3
EXAMPLE                        NO           3
APEX_1701140435539813          NO           3
MIKE_PLAYS_WITH_TDE            YES          3

7 rows selected.

Ah ... so my new tablespace is encrypted. Not bad ... so far TDE has no influence. I can create objects in this tablespace, query them etc. It is not disturbing at all. Good.
.

How does this key thing work in the DBaaS Cloud?

The documentation in above WP tells us this:

Managing the Software Keystore and Master Encryption Key

Tablespace encryption uses a two-tiered, key-based architecture to transparently encrypt (and decrypt) tablespaces. The master encryption key is stored in an external security module (software keystore). This master encryption key is used to encrypt the tablespace encryption key, which in turn is used to encrypt and decrypt data in the tablespace.

When the Database as a Service instance is created, a local auto-login software keystore is created. The keystore is local to the compute node and is protected by a system-generated password. The auto-login software keystore is automatically opened when accessed.

You can change (rotate) the master encryption key by using the tde rotate masterkey  subcommand of the dbaascli  utility. When you execute this subcommand you will be prompted for the keystore password. Enter the password specified when the service instance was created.
.

Creating a new PDB

That's easy, isn't it?

SQL> alter session set container=cdb$root;

Session altered.

SQL> create pluggable database pdb2 admin user mike identified by mike;

Pluggable database created.

SQL> alter pluggable database pdb2 open;

Pluggable database altered.

SQL> select tablespace_name, encrypted, con_id from CDB_TABLESPACES order by 3;

TABLESPACE_NAME                ENC     CON_ID
------------------------------ --- ----------
SYSTEM                         NO           4
SYSAUX                         NO           4
TEMP                           NO           4
USERS                          NO           4

SQL> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_s
ystem_cbn8fo1s_.dbf

/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_s
ysaux_cbn8fo20_.dbf

/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_u
sers_cbn8fo27_.dbf

Ah, bad thing. As I neither used the file_name_convert option nor changed the PDB_FILE_NAME_CONVERT initialization parameter my new PDB files get created in the "root" path of the CDB. I don't want this. But isn't there this cool new feature called ONLINE MOVE OF DATAFILES in Oracle Database 12c? Ok, it's an EE feature but let me try this after checking the current OMF file names in DBA_DATA_FILES and DBA_TEMP_FILES:

SQL> !mkdir /u02/app/oracle/oradata/MIKEDB/PDB2

SQL> ALTER DATABASE MOVE DATAFILE '/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_system_cbn8fo1s_.dbf' TO '/u02/app/oracle/oradata/MIKEDB/PDB2/system01.dbf';

SQL> ALTER DATABASE MOVE DATAFILE '/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_sysaux_cbn8fo20_.dbf' TO '/u02/app/oracle/oradata/MIKEDB/PDB2/sysaux01.dbf';

SQL> ALTER DATABASE MOVE DATAFILE '/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_users_cbn8fo27_.dbf' TO '/u02/app/oracle/oradata/MIKEDB/PDB2/users01.dbf';

Be prepared:
This will create a 1:1 copy of the file in the designated location and synch afterwards. It may take a minute per file.

And moving the TEMP tablespace(s) file(s) will fail.

SQL> ALTER DATABASE MOVE DATAFILE '/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_temp_cbn8fo25_.dbf' TO '/u02/app/oracle/oradata/MIKEDB/PDB2/temp01.dbf';

*
ERROR at line 1:
ORA-01516: nonexistent log file, data file, or temporary file
"/u02/app/oracle/oradata/MIKEDB/2A6680A0D990285DE053BA32C40AED53/datafile/o1_mf_temp_cbn8fo25_.dbf"

The temporary tablespace will have to be dropped and recreated. But not a big deal. 

Check;

SQL> select file_name from dba_data_files;

FILE_NAME
-----------------------------------------------------
/u02/app/oracle/oradata/MIKEDB/PDB2/sysaux01.dbf
/u02/app/oracle/oradata/MIKEDB/PDB2/users01.dbf
/u02/app/oracle/oradata/MIKEDB/PDB2/system01.dbf

Let me fix this so I don't hit this pitfall again:

SQL> alter system set pdb_file_name_convert='/u02/app/oracle/oradata/MIKEDB/pdbseed','/u02/app/oracle/oradata/MIKEDBPDB2';

Final verification:

SQL> select name, value from v$system_parameter where con_id=4;

NAME                   VALUE
---------------------- ----------------------------------
resource_manager_plan
pdb_file_name_convert  /u02/app/oracle/oradata/MIKEDB/pdbseed, /u02/app/oracle/oradata/MIKEDBPDB2


Now the fun part starts ... ORA-28374: typed master key not found in wallet

Remember this command from above in my PDB1? It run fine. But now it fails in PDB2.

SQL> create tablespace MIKE_PLAYS_WITH_TDE datafile '/u02/app/oracle/oradata/MIKEDB/PDB2/mike_plays_with_tde.dbf' size 10M;

create tablespace MIKE_PLAYS_WITH_TDE datafile '/u02/app/oracle/oradata/MIKEDB/PDB2/mike_plays_with_tde.dbf' size 10M
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

Voodoo in the database? I'm worried - especially as Roy had the same issue days before. But why did the command pass through without issues before in PDB1 - and now it doesn't in PDB2? Is it because the PDB1 is precreated - and my PDB2 is not?

Kinda strange, isn't it?
So connecting back to my PDB1 and trying again: 

SQL> alter session set container=pdb1;

Session altered.

SQL> create tablespace MIKE_STILL_PLAYS_WITH_TDE datafile '/u02/app/oracle/oradata/MIKEDB/PDB1/mike_still_plays_with_tde.dbf' size 10M;

Tablespace created.

Ok, now I'm worried.
What is the difference between the precreated PDB1 and my new PDB2?
.

Why do I get an ORA-28374 in my fresh PDB2?

When we compare the wallet status in both PDBs we'll recognize the difference:

PDB1:

SQL> select wrl_type, wallet_type, status from v$encryption_wallet;

WRL_TYPE        WALLET_TYPE          STATUS
--------------- -------------------- -----------------------
FILE            AUTOLOGIN            OPEN

PDB2:

SQL> select wrl_type, wallet_type, status from v$encryption_wallet;

WRL_TYPE        WALLET_TYPE          STATUS
--------------- -------------------- -----------------------
FILE            AUTOLOGIN            OPEN_NO_MASTER_KEY

.
Now thanks to Brian Spendolini I have a working solution. But I'm not convinced that this is an obvious path ...
Remember? I just would like to create a tablespace in my new (own, non-precreated) PDB. That's all ... 

SQL> alter session set container=cdb$root;

SQL> administer key management set keystore close;

keystore altered.

SQL> administer key management set keystore open identified by <your-sysadmin-pw> container=all;

keystore altered.

SQL> alter session set container=pdb2;

Session altered.

SQL> SELECT WRL_PARAMETER,STATUS,WALLET_TYPE FROM V$ENCRYPTION_WALLET;

WRL_PARAMETER                             STATUS             WALLET_TYPE
----------------------------------------- ------------------ -----------
/u01/app/oracle/admin/MIKEDB/tde_wallet/  OPEN_NO_MASTER_KEY PASSWORD

SQL>  ADMINISTER KEY MANAGEMENT SET KEY USING TAG  "tde_dbaas" identified by <your-sysadmin-pw>  WITH BACKUP USING "tde_dbaas_bkup"; 

keystore altered.

SQL> SELECT WRL_PARAMETER,STATUS,WALLET_TYPE FROM V$ENCRYPTION_WALLET;

WRL_PARAMETER                             STATUS   WALLET_TYPE
----------------------------------------- -------- -----------
/u01/app/oracle/admin/MIKEDB/tde_wallet/  OPEN     PASSWORD

And finally ... 

SQL> create tablespace MIKE_STILL_PLAYS_WITH_TDE datafile '/u02/app/oracle/oradata/MIKEDB/PDB2/mike_still_plays_with_tde.dbf' size 10M;

Tablespace created.

Wow!!!

That was not straight forward at all. Maybe it all happens due to my almost non-existing knowledge about TDE.

Ah ... and let me say that I find the missing uppercase letter with all keystore altered. echo messages quite disturbing. But this is a generic one and non-critical of course ...

--Mike
.

PS: Read on about Seth Miller's experience here on Seth's blog:
http://sethmiller.org/oracle-2/oracle-public-cloud-ora-28374-typed-master-key-not-found-in-wallet/ 




Tuesday Jan 19, 2016

Clean up APEX - Journey to the Cloud IV

DBaaS Oracle Cloud

What happened so far on my Journey to the Cloud?

Today's journey: Cleanup APEX removal leftovers 

When you read my "Patch" blog post from Dec 22, 2015 you'll see that I was left with an incomplete Oracle Application Express (APEX) removal situation. Something looked really strange and didn't work as expected and documented. 

But thanks to Jason Straub and Joel Kallman, my APEX colleagues, The solution below got provided by Jason. And let me say upfront that it's not APEX's fault ;-) Somebody has mixed up stuff in the current DBaaS deployment and thus the correct scripts to remove things in the right order are simply missing.

How to remove Oracle APEX from the DBaaS Cloud's CDB$ROOT

  1. Download APEX 4.2.0.6 and PDBSS 2.0 (Multitenant Self Service Provisioning Application):
    http://www.oracle.com/technetwork/developer-tools/apex/application-express/apex-archive-42-1885734.html
    http://www.oracle.com/technetwork/database/multitenant/downloads/multitenant-pdbss-2016324.html

    Actually if you have removed PDBSS already as I did in my "Patch" blog post from Dec 22, 2015 you don't have to download, unzip and execute the PDBSS removal script again.


  2. Copy the zip files to the Cloud environment

    scp -i ./yourCloudKey /media/sf_CTEMP/Download/apex_4.2.6_en.zip oracle@<your_cloud_IP>:/home/oracle

    Enter passphrase for key './yourCloudKey':

    apex_4.2.6_en.zip              100%   82MB 116.9KB/s   11:58

    Repeat the same with PDBSS in case you didn't remove it already.


  3. Connect to your Cloud environment and unzip both files.

    ssh -i ./yourCloudKey oracle@<your_cloud_IP>

    cd
    unzip apex_4.2.6_en.zip


    Repeat the same with PDBSS in case you didn't remove it already.


  4. Remove PDBSS by using your unzipped archive

    cd /home/oracle/pdbss
    sqlplus / as sysdba
    SQL> @pdbss_remove.sql


  5. Remove APEX 5 by using the existing default installation
    (this is the part I did already in my previous blog post from Dec 22)

    cd $ORACLE_HOME/apex
    sqlplus / as sysdba
    SQL> @apxremov.sql 



  6. Remove APEX 4.2 parts by using your unzipped archive
    (this is the part I didn't do yet)

    cd /home/oracle/apex
    sqlplus / as sysdba
    SQL> @apxremov_con.sql


  7. Drop the DBaaS Monitor common users

    As the DBaaS Monitor is based on APEX as well removing APEX will "kill" the DBaaS Monitor as well. So in oder to have a proper working environment you must drop the common users for DBaaS Monitor as well. Otherwise you'll receive ORA-959 synch errors in PDB_PLUG_IN_VIOLATIONS whenever you deploy a new PDB and - even worse - the PDB won't open unrestricted.

    SQL> drop user C##DBAAS_BACKUP cascade;
    SQL> drop user C##DBAAS_MONITOR cascade;


  8. Finally recompilation and check for invalid objects

    sqlplus / as sysdba
    SQL> @?/rdbms/admin/utlrp.sql

    SQL> select object_name, owner from dba_objects where status='INVALID';
    no rows selected


  9. Now you are free to install APEX 5.0 in the PDBs of your choice.

    As theres's a PDB1 already I will create APEX inside this PDB first.

    cd $ORACLE_HOME/apex
    sqlplus / as sysdba
    SQL> alter session set container=pdb1;
    SQL> create tablespace APEX5 datafile '
    /u02/app/oracle/oradata/CDB1/PDB1/apex5_01.dbf' size 100M autoextend ON next 1M;


    Be aware: PDB_FILE_NAME_CONVERT is not set. And your database is not in ASM. Therefore avoiding the data file path and name will let you end up with a file named after OMF standards in:
    /u02/app/oracle/oradata/CDB1/26A65D56D16F21A1E05336036A0A1AD8/datafile/o1_mf_apex5_c9wg9kly_.dbf

    Create APEX 5.0 in PDB1, change the admin password, create the APEX listener and REST service and unlock the public user:

    SQL> @apexins.sql APEX5 APEX5 TEMP /i/
    SQL> @apxchpwd.sql


    Be aware about the password requirements:

    --------------------------------------------------------------------------------
    Password does not conform to this site's password complexity rules.
    * Password must contain at least 6 characters.
    * Password must contain at least one numeric character (0123456789).
    * Password must contain at least one punctuation character
    (!"#$%&()``*+,-/:;?_).
    * Password must contain at least one upper-case alphabetic character.
    * Password must not contain username.
    --------------------------------------------------------------------------------


    SQL> @apex_rest_config.sql
    SQL> alter user APEX_PUBLIC_USER identified by SecretPWD account unlock;

Let me add that it is possible to have different APEX versions in different PDBs. The only thing which you'll need to really really take care on is the images directory and the different apex homes.

That's it ... and in a future cloud deployment the extra step to remove APEX 4.2 shouldn't be necessary anymore. 

--Mike

Tuesday Dec 22, 2015

Patch, patch, patch - Journey to the Cloud - III

DBaaS Oracle Cloud

What happened so far on my Journey to the Cloud?

I haven't stopped my journey if you wonder about the inactivity for two weeks. I just had to learn a bit as my pretty naive approach caused me some trouble. Based on my last experience I had to find out why APEX left so many leftovers when I removed it the friendly way. And I wanted to patch my environment to a higher version of PSUs as - learning experience - a new deployment does not get deployed with the most recent PSU applied from scratch.  So there's still plenty of stuff to do.
.

Patching my DBaaS database with the most recent PSU

If you've got frustrated in the past by yourself while applying a PSU then please read on - I guarantee you'll smile more than once ...

First of all what is my expectation from a cloud environment?
Yes, Push A Button.

Well ... with the July 2015 PSU (12.1.0.2.4) this worked just fine even though it took a bit of time to download the PSU. But it got applied flawlessly by just pushing a button.

PSU Choice

But we have December 2015 already. So I would like to apply the Oct 2015 PSU (12.1.0.2.5) to my environment. And it turns out that currently this is the only one getting offered in the DBaaS environment to be applied. 

First step: Execute the precheck

See the results ...

Oh, it failed. But why? It does not tell me anything about the reason why it had failed. Was it no space left? Or a patch conflict? Or something else? No idea as Error [1] isn't very useful.  

And what's next? Live without the PSU even though I'm preaching since ages in our workshops that you MUST apply PSUs on a regular basis? Ok, so I have to sort this out.
,

Let's find it out

I did login to my environment via SSH. Then:

cd /u01

Ah, and there's a psu.n subdirectory - that looks promising. And there's a file conflictlog in it - that does look even more promising. 

It tells me:

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-47: The patch(es) has supersets with other patches installed in the Oracle Home (or) among themselves.

ZOP-40: The patch(es) has conflicts with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

Summary of Conflict Analysis:
There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
21359755, 21627366

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
20281121

Whole composite patch Conflicts/Supersets are:
Composite Patch : 21359755

        Conflict with 21627366
        Bug Superset of 20281121

Detail Conflicts/Supersets for each patch are:

Sub-Patch : 21359755
        Conflict with 21627366
        Conflict details:
        /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a:kzan.o
        /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a:kspt.o
        /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a:kcb.o
        /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a:kcrfw.o
        /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a:kokt.o

        Bug Superset of 20281121
        Super set bugs are:
        20281121

Patch failed with error code 1

Ah, my friend Error [1] again. Now all is clear, isn't it? There's a conflict as the previous installation deployed in the cloud does seem to have gotten some extra treatments - which is good in one way but bad in another as I will have to solve this now. The Cloud Console does not offer me anything to solve this.
.

Luckily ...

I'm subscribed to our internal cloud mailing lists. And other people are way more smarter than I am so I found an email linking an explanation in the official documentation (Known Issues for the Database Cloud As A Service). There a quite a few known issues and it's very useful to have such a document. And here we go with the solution to my problem:

Ok, I have two options, one in the graphical interface, the other one on the command line. I'll go with the first option as this is meant to be Push A Button style and not type on the command line.

So first I click on PATCH in the hamburger menu: 

And then I chose the FORCE option.
May the force be with me.

The alternative would have been on the command line using the dbpatchm subcommand of the dbaascli utility:

Before applying the patch, set the value of the ignore_patch_conflict key to 1 in the /var/opt/oracle/patch/dbpatchm.cfg patching configuration file; for example:

ignore_patch_conflict=1
.

Et voilà ...

I took a bit ...

Actually a bit over 30 minutes ... but finally ...

The most recent PSU 12.1.0.2.5 from October 2015 has been applied to my DBaaS Cloud installation. It wasn't that complicated - if I had known upfront to hit the magic "FORCE" button ;-)

Finally let's check:

COLUMN PATCH_ID FORMAT 99999999
COLUMN PATCH_UID FORMAT 99999999
COLUMN VERSION FORMAT A12
COLUMN STATUS A12
COLUMN DESCRIPTION A30

select  PATCH_ID, PATCH_UID, VERSION, STATUS, DESCRIPTION from DBA_REGISTRY_SQLPATCH order by  BUNDLE_SERIES;


.

Remove APEX from my database and install it into my PDB

In my previous journey removing APEX from my cloud database didn't work quite well. I had leftovers afterwards, mainly from the Multitenant Self Service Provisioning APEX application (owner: C##PDBSS) and from the Database-As-A-Service-Monitor. 

Removing  Multitenant Self Service Provisioning Application

From some internal email conversation (and from the readme.pdf included in the download of the Multitenant Self Service Provisioning Application) I learned that there's a pdbss_remove.sql. And a find showed me the unusual location in the cloud deployment:

/var/opt/oracle/log/pdbss/pdbss

So first of all I connected to my CDB$ROOT and started the removal script:

sqlplus / as sysdba
spool /tmp/pdbss_remove.log
@/var/opt/oracle/log/pdbss/pdbss/pdbss_remove.sql

It took 1:45 minutes to complete. Spool off is not necessary as the script ends SQL*Plus.

Then I started the APEX removal script from the $ORACLE_HOME/apex subdirectory:

cd $ORACLE_HOME/apex
sqlplus / as sysdba

@apxremov_con.sql

Well ... but again something seemed to fail as I end up with a good bunch of invalid objects. 

First I did recompile:

@?/rdbms/admin/utlrp.sql 

But even then those 3 objects kept INVALID after the removal.

SQL> select owner, object_name from dba_objects where status='INVALID';

OWNER         OBJECT_NAME
------------- --------------------------
SYS           WWV_DBMS_SQL
FLOWS_FILES   WWV_BIU_FLOW_FILE_OBJECTS
APEX_040200   APEX$_WS_ROWS_T1

And furthermore a good number of APEX user schema did not get removed as well.

CON_ID USERNAME
------ --------------------------
     1 APEX_REST_PUBLIC_USER
     1 APEX_PUBLIC_USER
     1 APEX_LISTENER
     1 APEX_040200
     3 APEX_REST_PUBLIC_USER
     3 APEX_PUBLIC_USER
     3 APEX_LISTENER
     3 APEX_040200

8 rows selected.

Will update this blog post as soon as I have more news about how to remove APEX flawless from the cloud deployment. The issue is under investigation.

So better don't remove APEX at the moment from the DBaaS cloud deployment. 

--Mike


Friday Dec 04, 2015

Switch on/off + Remove - Journey to the Oracle Cloud - II

DBaaS Oracle Cloud

What happened so far on my Journey to the Cloud?

Day 2 of my Cloud exploration journey.

In my blog posts about my DBaaS Cloud journey I won't explain a lot about how to login to the Oracle Cloud or how to setup a simple database as others have done this already including the SSH key topic:

But let me show you some findings I've had when logging in via SSH to my DBaaS environment. In the following article I will describe how to:

  • Switch off Flashback Database
  • Clean up expired Backups and Archivelogs
  • Clean up traces and set the retention policy
  • Remove APEX from the CDB$ROOT
.

Flashback Database is ON by default

After playing around with my DBaaS environment for a few days all got stuck. The database hung. A quick look into the alert.log revealed the simple reason: my database had no space anymore on the Flash Recovery Area mount point. "Why that?" I did ask myself.

Well,

SQL> select FLASHBACK_ON from V$DATABASE;
YES

told me the answer. By default FLASHBACK DATABASE is ON. Great idea but for playground-mode you may or may not switch it OFF. Furthermore in my (developers) environment the logs will be written to /u03/app/oracle/flash_recovery_area/flashback (DB_RECOVERY_FILE_DEST) - and DB_RECOVERY_FILE_DEST_SIZE is defined at only 7.5GB. That got consumed quickly.  

You may switch it OFF if you don't need FLASHBACK DATABASE at the moment. Or check at least increase the DB_RECOVERY_FILE_DEST_SIZE.

SQL> alter database flashback off; 
SQL> exit
$  rm /u03/app/oracle/flash_recovery_area/flashback/*

And just for the records:
You don't need to have FLASHBACK DATABASE switched ON as a feature in order to use GUARANTEED RESTORE points (which is one of the best things - I used it almost daily before I "destroy" something in my databases).

Quick check:

SQL> select FLASHBACK_ON from V$DATABASE;
NO

SQL> create restore point DESTROY_ME guarantee flashback database;

If you'd check now:

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------------
RESTORE POINT ONLY

Don't forget to drop your GUARANTEED RESTORE POINT later on. 

SQL> drop restore point DESTROY_ME; 

.

Clean up your expired BACKUPS and ARCHIVE LOGS 

Every database should be run in archivelog mode - no doubt. But if you are short on disk space for the archive logs you may need to clean up from time to time. Furthermore archivelog mode is necessary in order to support certain actions in Oracle Multitenant.

Keep in mind:
Backups and Archivelogs don't clean up themselves.

Clean up expired backups:

$ rman target /
RMAN> crosscheck backup;
RMAN> delete noprompt obsolete;
RMAN> delete expired backup;

Clean up archivelogs: 

$ rman target /
RMAN> crosscheck archivelog all;
RMAN> delete noprompt expired archivelog all;


Clean up Trace and Log Files and set the Retention

The default retention for traces has been set to standard values in the cloud as well. You may change this to get a more frequent purging of traces. The interface to tweak settings is ADRCI. Find a nice ADRCI Survival Guide for DBAs here written by my colleague Uwe Hesse, 

adrci> show control

To solve DIA-48448 you'll have to set the correct diag directory you want to look at:

adrci> show homes
ADR Homes:
diag/rdbms/mike/MIKE
diag/tnslsnr/slc01qnv/listener
diag/tnslsnr/MIKEVM/listener

adrci> set home diag/rdbms/mike/MIKE

adrci> show control

ADR Home = /u01/app/oracle/diag/rdbms/mike/MIKE:
*************************************************************************
ADRID                SHORTP_POLICY        LONGP_POLICY         LAST_MOD_TIME                            LAST_AUTOPRG_TIME                        LAST_MANUPRG_TIME                        ADRDIR_VERSION       ADRSCHM_VERSION      ADRSCHMV_SUMMARY     ADRALERT_VERSION     CREATE_TIME
-------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- ---------------------------------------- -------------------- -------------------- -------------------- -------------------- ----------------------------------------
4162796853           720                  8760 2015-09-17 13:51:04.699486 +00:00        2015-11-30 17:53:33.062134 +00:00                                                 1                    2                    82                   1                    2015-09-17 13:51:04.699486 +00:00
1 rows fetched

This means SHORTP_POLICY and LONGP_POLICY are set to 30 days and 365 days (~1 year). 

I'd usually change it (that is pretty up to your personal preferences) and purge whatever is older than a few days for a start.

adrci> set control (SHORTP_POLICY=96, LONGP_POLICY=720)

switching the short term policy to 4 days, the long term policy (alert.log, incidents) to 30 days. And finally cleaning up also older traces for a fresh start:

adrci> purge -age 24

This tool a while (~ 10 minutes) in my environment.
.

Remove Oracle Application Express (APEX) from the CDB$ROOT

As I wrote a few weeks ago Oracle APEX is a fantastic - and unfortunately still the most undervalued - Oracle tool. The only real treatment you should do to your Cloud database first is removing APEX from the CDB$ROOT before either plugging something in or creating pluggable database. If you have created pluggable databases (PDBs) already please make sure that APEX is not used in any of them as removing APEX from the CDB$ROOT will remove it from all existing PDBs as well.

See: https://blogs.oracle.com/UPGRADE/entry/apex_in_pdb_does_not 
on Why and How to remove Oracle APEX from the CDB$ROOT.

In short:

  1. Make sure to change into the local $ORACLE_HOME/apex directory first before starting SQL*Plus:
    cd $ORACLE_HOME/apex

  2. Connect to your CDB$ROOT:
    sqlplus / as sysdba

  3. Run the "Remove APEX from the CDB$ROOT" script:
    SQL> @apxremov_con.sql

  4. Recompile to get rid of any invalid objects left over by the removal script:
    SQL> @?/rdbms/admin/utlrp.sql

That sounds simple, doesn't it?

Well, but my journey ends here for today as I'm getting 28 invalid objects (which don't compile) afterwards - and plenty of nice ORA-600 errors in the alert.log looking fantastically similar to Data Dictionary corruptions. I think my Part 3 of the journey will be a patching cycle. 

ORA-00600: internal error code, arguments: [ktsircinfo_num1], [1], [4], [71210]
ORA-00600: internal error code, arguments: [6711], [4228381], [1], [4228380], [3
]

A look into the list of INVALID objects give me this result - from the Cloud documentation I see that C##PDBMGR is a self-provisioning user env:

The Oracle Pluggable Database Self-Service Provisioning application provides an interface to Oracle Database 12c Multitenant option and allows for the provisioning of Pluggable Databases (PDBs). You can perform PDB operations including create, clone, open/close, plug/unplug, and drop. Additionally, you can grant others access to the application, giving them rights to create and manage their own PDBs within the quota limits you set.

So maybe it is not a good idea to remove APEX from the CDB$ROOT in our Oracle DBaaS Cloud environment? I will find out next week ...

SQL> column object_type heading "Type" Format a15
SQL> column object_name heading "Name" format a30
SQL> column owner heading "Owner" format a20
SQL> set pagesize 100
SQL> select owner, object_type, object_name from dba_objects where status='INVALID';

Owner                Type            Name
-------------------- --------------- ------------------------------
SYS                  PACKAGE BODY    WWV_DBMS_SQL
FLOWS_FILES          TRIGGER         WWV_BIU_FLOW_FILE_OBJECTS
APEX_040200          TRIGGER         APEX$_WS_ROWS_T1
C##DBAAS_MONITOR     FUNCTION        DBM_IS_CDB
C##PDBMGR            PACKAGE BODY    PDBSS_PDB_MGR
C##PDBMGR            TRIGGER         PDBSS_APPLICATION_LOG_BI
C##PDBMGR            TRIGGER         PDBSS_ERROR_LOG_BI
C##PDBMGR            TRIGGER         PDBSS_PREFERENCES_BIU
C##PDBMGR            TRIGGER         PDBSS_RESERVED_DB_NAMES_BIU
C##PDBMGR            TRIGGER         PDBSS_DB_SIZES_BIU
C##PDBMGR            TRIGGER         PDBSS_RESOURCE_PLANS_BIU
C##PDBMGR            TRIGGER         PDBSS_USERS_BIU
C##PDBMGR            TRIGGER         PDBSS_USER_LOG_BI
C##PDBMGR            TRIGGER         PDBSS_ACCESS_REQUESTS_BIU
C##PDBMGR            TRIGGER         PDBSS_DATABASES_BIU
C##PDBMGR            TRIGGER         PDBSS_DATABASE_LOG_BI
C##PDBMGR            TRIGGER         PDBSS_DATABASE_ADMINS_BIU
C##PDBMGR            TRIGGER         BIU_PDBSS_EMAIL_TEMP_TYPES
C##PDBMGR            TRIGGER         BIU_PDBSS_EMAIL_TEMPLATES
C##PDBMGR            PACKAGE         PDBSS_FILTER_FW
C##PDBMGR            PACKAGE         PDBSS_FW
C##PDBMGR            TRIGGER         PDBSS_HISTORY_BIU
C##PDBMGR            TRIGGER         PDBSS_NOTIF_BIU
C##PDBMGR            TRIGGER         PDBSS_TZ_PREF_BIU
C##PDBMGR            PACKAGE BODY    PDBSS
C##PDBMGR            PACKAGE BODY    PDBSS_FILTER_FW
C##PDBMGR            PACKAGE BODY    PDBSS_FW
C##PDBMGR            PACKAGE BODY    PDBSS_EMAIL_API

28 rows selected.

I certainly can drop the user C##PDBMGR - but removing the Database Monitor user C##DBAAS_MONITOR may cause some trouble as the DBaaS Monitor won't work anymore. And furthermore still three APEX objects remain INVALID.

I'll update you as soon as I have my environment patched and refreshed sometime next week after UKOUG Conference.

--Mike

Thursday Dec 03, 2015

Push a Button - Journey to the Oracle Cloud - I

DBaaS Oracle Cloud

What happened so far on my Journey to the Cloud?

Isn't that a great marketing picture/poster?

But the real question is:

  • How do you move a database to the Oracle Cloud?
  • Is it easy or complicated?
  • What is your benefit?
  • Which techniques work, which don't?
  • And what should you switch on/off or tweak?

I'm not (yet) a cloud expert. But my goal for upcoming posts in the next weeks will be to show you my naive approach of moving databases to the Oracle Cloud and get some true benefit from it.

--Mike

About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle

Based in Germany. Interlink between customers/partners and the Upgrade Development. Running workshops between Arctic and Antartica. Assisting customers in their reference projects onsite and remotely. Connect via:

- -

Search


Archives
« February 2017
SunMonTueWedThuFriSat
   
2
4
5
6
7
10
11
12
14
15
18
19
22
23
24
25
26
27
28
    
       
Today
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
Workshop Map
x Oracle related Tech Blogs
This week on my Rega & Pono
Upgrade Reference Papers