Thursday Jan 14, 2016

VBox 5.0.10/12 issues with PERL and Seg Faults - UPDATE

A bit more than two months ago I did hear from several people having issues with our Hands-On Lab environment. And it became clear that only those who use Oracle Virtual Box 5 see such errors. 

Then I read Danny Bryant's blog post (thanks to Deiby Gomez for pointing me to it) about similar issues and a potential solution yesterday:

And interestingly one of my colleagues, our PL/SQL product manager Bryn Llewellyn started an email thread and a test internally yesterday as well. The issue seem to occur only on newer versions of Apple's MacBooks.
.

Potential Root Cause

The PERL issues seem to happen only on specific new Intel CPUs with a so called 4th level cache.

The current assumption is that Intel CPUs with Iris Pro graphics are affected. Iris Pro means eDRAM (embedded DRAM) which is reported as 4th level cache in CPUID. We have confirmed that Crystal Well and Broadwell CPUs with Iris Pro are affected. It is likely that the Xeon E3-1200 v4 family is also affected.

It seems to be that there's a bug in the perl binary. It links against ancient code from the Intel compiler suite doing optimizations according to the CPU features. Very recent Intel CPUs have 4 cache descriptors.

People who encountered this used Virtual Box VBox 5.0.x - and it passes this information to the guest. This leads to a problem within the perl code. You won't see it on VBox 4.3 as this version does not pass the information to the guest. 

But actually it seems that this issue is independent of Virtual Box or any other virtualization software. It simply happens in this case as many people use VBox on Macs - and some Macs are equipped with this new CPU model. But people run Oracle in VBox environments and therefore see the issue as soon as they upgraded to VBox 5.0.x.
.

Potential Solutions

If you are using Oracle in VBox there are actually two solutions:

  • Revert to VBox 4.3 as this won't get you in trouble
    This problem was not triggered on VBox 4.3.x because this version did not  pass the full CPUID cache line information to the guest.
    .
  • Run this sequence of commands in VBox 5.0 to tweak the CPUID bits passed to the guest:
    .
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/Leaf" "0x4"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/SubLeaf" "0x4"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/eax"  "0"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/ebx" "0" 
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/ecx" "0" 
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/edx"  "0"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/SubLeafMask" "0xffffffff" 

    • Of course you'll need to replace VM_NAME by the name of your VM
      .

If the error happens on a bare metal machine meaning it happens not inside a virtual image but on a native environment then the only chance you'll have (to my knowledge right now) is to exchange the PERL before doing really something such as running root.sh or rootupgrade.sh in your Grid Infrastructure installation or before using the DBCA or the catctl.pl tool to create or upgrade a database.

In this case please refer to the blog post of Laurent Leturgez:

Issues with Oracle PERL causing segmentation faults:

http://laurent-leturgez.com/2015/05/26/oracle-12c-vmware-fusion-and-the-perl-binarys-segmentation-fault

.

Further Information

This issues is currently tracked internally as bug 22539814: ERRORS INSTALLING GRID INFRASTRUCTURE 12.1.0.2 ON INTEL CPUS WITH 4 CACHE LEVEL.

So far we have not seen reports by people encountering this in a native environment but only by people using VBox 5.0.x or Parallels or VMware on a very modern version of Apple hardware.

.
--Mike 



Wednesday Dec 16, 2015

New version of our Hands-On Lab available for download "Upgrade/Migrate/Consolidate to Oracle 12c" HOL

Yes! Finally ...

Roy did all the hard work and patched all our environments with the most recent PSUs. And he managed also to upload our Hands-On Lab not only internally to retriever (Oracle internally) but also to OTN.

What you'll be able to do?

  • Upgrade an 11.2.0.4 database UPGR to Oracle Database 12.1.0.2
  • Plug in UPGR into the CDB2, an Oracle Database 12.1.0.2 Container database, and make it a PDB
  • Migrate an 11.2.0.4 database FTEX into another PDB using the Full Transportable Export/Import feature
  • Unplug an Oracle Database 12.1.0.1 PDB and plug it into an Oracle Database 12.1.0.2 container database and finally upgrade it
  • And of course you can play around with the entire environment
    .

What we've did for your convenience? 

  • Switched to a new PDF viewer which allows better copy&paste - see the instructions document.
    Unfortunately we are not allowed to distribute the relatively old version of Adobe PDF Reader for Linux. 
  • Applied the most recent PSUs to the 11.2.0.4 and 12.1.0.2 homes
  • Fixed some minor things
  • Updated the HOL Instructions
  • Added an "aritificial" environment variable $OH12 to shorten typing
  • And some other tiny little things ...
  • And the lab is packaged with Oracle VBox 5.0.10 including the matching VBox Extensions which caused us trouble with the previous lab environment which was settled on Vbox 4.3 - so we hope that now all will go well.
    .

Where can you download the lab and the instructions?

All credits to Roy!!!

Thanks
--Mike

Thursday Nov 12, 2015

Oracle VirtualBox 5.0.x - Segmentation Fault in PERL

Please see also:

 


 

Yesterday and the day before I've exchanged several emails with Ana who downloaded our Hands-On-Lab from here:

after OOW15, encountering a SEGMENTATION FAULT when trying to start the database upgrade with catctl.pl:

$ $ORACLE_HOME/perl/bin/perl catctl.pl catupgrd.sql
Segmentation fault 

Very strange thing ... 

The database is in upgrade mode (checked this in the alert.log) and there are no strange things mentioned anywhere. Plus hundreds of people have run and completed our lab so far.

Tue Nov 10 20:39:47 2015
MMON started with pid=21, OS id=9828
Starting background process MMNL
Tue Nov 10 20:39:47 2015
MMNL started with pid=22, OS id=9832
Stopping Emon pool
Tue Nov 10 20:39:47 2015
ALTER SYSTEM enable restricted session;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET _system_trig_enabled=FALSE SCOPE=MEMORY;
Autotune of undo retention is turned off.
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET _undo_autotune=FALSE SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET undo_retention=900 SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET enable_ddl_logging=FALSE SCOPE=MEMORY;
Resource Manager disabled during database migration: plan '' not set
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET resource_manager_plan= SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET recyclebin='OFF' DEFERRED SCOPE=MEMORY;
Resource Manager disabled during database migration
replication_dependency_tracking turned off (no async multimaster replication found)
AQ Processes can not start in restrict mode
Starting background process CJQ0
Tue Nov 10 20:39:47 2015
CJQ0 started with pid=27, OS id=9836
Completed: ALTER DATABASE OPEN MIGRATE 

We checked several other things - and then I came across this tweet by Martin Klier yesterday:

and started to search a bit

I have no 100% proof for the actual reason but several people seem to have issues with SEGMENTATION FAULTs in Oracle's PERL ($ORACLE_HOME/perl/bin/perl) when using Oracle VirtualBox 5.0.x - and according to VitualBox Forum that seems to happen with the most recent VBox 5.0.10 as well.

The "funny" thing is that all works perfectly well in VBox 4.3.x ...  

It reminds me a lot on the reoccuring VBox bug with my German keyboard not allowing me to type in the | (pipe) character which requires to press "ALT GR" +  "<" keys together.  

--Mike

.

Monday Oct 26, 2015

OOW 2015 - if you've missed our Upgrade Hands On Lab

Ready to kick off with the first of our four Hands-On-Labs at OOW15 at the Nikko Hotel. Currently it's calm before the storm  ...

OOW 2015

But just in case you didn't grab a seat at one of the labs then please come to our talk today (Monday!) at 1:30pm in Moscone South 102 as we'll show you the core parts of the lab - and you'll be able to download it later on from the blog and run it by yourself.

Download the lab via this link:

--Mike

Friday Oct 02, 2015

OOW 2015 Sessions and Labs - Oracle Open World

OOW 2015OMG ... only a few weeks to go ... Oracle Open World 2015 in San Francisco is coming closer and closer ...

And this year will be really tough as we have a reduced number of people there - but more work to do as in previous years. 3 talks (2 for Upgrade, 1 for Data Pump), 4 labs (all in Nikko Hotel 15min walking distance from Moscone Center) - plus a good number of customer meetings already lined up. Plus the chance to meet so many great people ... and not to forget the Data Warehouse Global Leaders event at the Oracle HQ. 

I have that strange feeling that I will be VERY tired when I'll board the plane on Friday night heading back to Germany ... ;-)
.

Focus On Upgrades/Migrations

As the fantastic application we are using for the OOW content catalog doesn't allow me to link directly to a session Roy has built a Focus On document to guide you to some important talks around Upgrades and Migrations at OOW2015.for your convenience: 


Talks

Session ID

Title

Start Time

Room

CON6777 Upgrade and Migrate to Oracle Database 12c: Live and Uncensored!

Many customers now have database environments numbering in the hundreds or even thousands. This session addresses the challenge of maintaining technical currency of such an environment while also containing upgrade and migration costs at a reasonable level. Learn from Oracle Database upgrade experts about product features, options, tools, techniques, and services that can help you maintain control of your database environment. You will also see examples of how real customers are successfully meeting this challenge today.
.
.
October 26
at 13:30h
Moscone South—102
CON8375 How to Upgrade Hundreds or Thousands of Databases in a Reasonable Amount of Time

Many customers now have database environments numbering in the hundreds or even thousands. This session addresses the challenge of maintaining technical currency of such an environment while also containing upgrade and migration costs at a reasonable level. Learn from Oracle Database upgrade experts about product features, options, tools, techniques, and services that can help you maintain control of your database environment. You will also see examples of how real customers are successfully meeting this challenge today.
.
.
October 28
at 12:15h
Moscone South—102
CON8376 Deep Dive: More Oracle Data Pump Performance Tips and Tricks

The Oracle Data Pump development team is back with even more performance tips and tricks for DBAs! In this session, learn about Oracle Data Pump features, parameters, and patches—some added since the first patch set of Oracle Database 12c 12.1.0.2—that will improve performance and decrease overhead for Oracle Data Pump projects. Whether you are an Oracle Data Pump novice or already an expert, you are sure to learn something new in this session that will help you maximize the throughput of your export and import operations.
.
.
October 29
at 9:30h

Moscone South—305

HOL10348 Upgrade, Migrate, and Consolidate to Oracle Database 12c [HOL10438]

The Oracle Data Pump development team is back with even more performance tips and tricks for DBAs! In this session, learn about Oracle Data Pump features, parameters, and patches—some added since the first patch set of Oracle Database 12c 12.1.0.2—that will improve performance and decrease overhead for Oracle Data Pump projects. Whether you are an Oracle Data Pump novice or already an expert, you are sure to learn something new in this session that will help you maximize the throughput of your export and import operations.
.
Oct 26 at 11:00h
Oct 27 at 11:45h
Oct 28 at 13:15h
Oct 29 at 12:30h

Hotel Nikko - Golden Gate


Hope to see you at OOW 2015!

--Mike

Tuesday Sep 09, 2014

OOW 2014 - Upgrade Hands-On-Lab at NIKKO HOTEL

This year's Oracle Open World will be a real challenge logistically for us. Non-privileged employees were assigned to airport hotels only. So Roy, Joe, Dean and I stay >16 miles (>25km) away from Moscone Center. And to make things even more complicated due to some reconstruction at the Moscone Center (I hope they'll get rid of their 1990s audio equipment and such) our Hands On Labs for instance are at the NIKKO Hotel, not at the Marriott Marquee anymore.

These are the dates for our HOL about How to upgrade to Oracle Database 12c and plug into Oracle Multitenant:

  • Monday , Sep 29 - 11:45 AM in Hotel Nikko - Bay View (0/32)
  • Tuesday , Sep 30 - 5:15 PM in Hotel Nikko - Bay View (0/32)
  • Wednesday, Oct 01 - 4:15 PM in Hotel Nikko - Bay View (0/32)
  • Thursday , Oct 02 - 8:30 AM in Hotel Nikko - Bay View (0/32)
Follow this link to the OOW Session Agenda - ID of the Labs is [HOL9127] - and usually you'll have to hurry as the 50 seats per lab sell out quickly.

Especially Thursday will be a tough call to be there at 8:30am in the morning. At least traffic on 101 won't be that terrible at such a night time ;-)

-Mike

Thursday Oct 10, 2013

OOW 2013: Recap - Thanks a lot!!!

OOW 2013 is over for a while now. And yesterday we've received the session results from our talk and the HOL. And we have to say Thank You Very Much! For your time, for the great discussions, for your feedbacks. This is the highest session count we've had in the past years.

Generally my overall feeling this year was that database sessions were very well attended - and the two sessions I wanted to see in my limited spare time were already sold out. And we've really had so many great discussion at our booth 007 - and Joe was an excellent "double" as Daniel Craig was kept up with other stuff ;-)


Joe Errede and Carol Tagliaferri

So thanks again - and we hope to make you the VBox image for the lab available soon for download. In between please feel free to download the slides from the download center - or simply grab the big deck about Upgrade to Oracle Database 12c.

Furthermore let me introduce the new sections with White Papers making it easier to find them and the new Oracle Blogs section with blogs I fequently visit.

And finally here are some impressions from OOW 2013:


Adam Levine from Maroon 5


Matt Flynn and Adam Levine from Maroon 5


At the "Friends of Pythian" event - thanks to Alex Gorbatchov for the invitation :-)


Yes - and OOW happens every year in SF :-)

Cheers - Mike

About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle

Based in Germany. Interlink between customers/partners and the Upgrade Development. Running workshops between Arctic and Antartica. Assisting customers in their reference projects onsite and remotely. Connect via:

- -

Search

Archives
« February 2016
SunMonTueWedThuFriSat
 
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
     
       
Today
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
Workshop Map
x Oracle related Tech Blogs
This week on my Rega & Pono
Upgrade Reference Papers