Gavin's Blog

Just a quick update for those of you that don't keep on eye on new blogs starting @ Oracle, Paul and the team at Oracle Certification have started blogging over here.

Should be an interesting blog to keep an eye on, especially for all those people that found my blog while looking to get Oracle Certified and needing encouragement. I know I've slackened off lately, but I'll be reading it to get back into the swing of getting certified. You should too.

Gavin

It all started on my second project for Oracle. I hadn't been in the job that long when I was put in charge of installing, patching and maintaining a bunch of different products for a new project here in Perth. It included three Portal installs, two Oracle Internet Directories, two Collaboration Suite installs and an eBusiness Suite and a few other bits and pieces (partridge in a pear tree...) . This would be the first of many environments (about 20!) we would be implementing, so we needed a some kind of layout or template to apply so it would be easy to find things and do multiple installs on one machine.

Having been a production DBA for some time, I knew we would be getting calls and emails from end users saying "I can't access portal" or "I need a patch applied to Apps". Not only did I need a way of identifying which portal they were talking about, but which machine was it installed on, what the admin port for enterprise manager was, username/password etc etc.

I had done some training with Oracle OnDemand just before the project, so I figured, who else would have a good set of standards, so I investigated how to install things "OnDemand Certified". Their standards were good, had a few pitfalls, but with a bit of extending, would work well. (The OnDemand standard at the time meant I couldn't install a second or third portal into the same environment - let alone the same server, not sure what they are at the moment).

OK, so how could I design and build these environments without a) confusing the heck out of me and my team, and b) killing one application by installing another one (try 2 ContentDB's on one machine and you'll know what I mean).

A few things were pretty apparent.
* Identifying an environment was first. No use getting an email saying Portal's down, then searching through 13 environments to find which one.
* Once you're in an environment, ports are a HUGE issue,
* Install directories are fairly important,
* While not 100% necessary, differing installation users and groups may help.

First, lets talk ports.

If you've ever done an install of Application Server, or CollabSuite, or eBS, (the list goes on), you'll know about ports and assigning port numbers to processes. I don't know if you've thought about it as you're clicking Next > Next > Next, but you can tell the install to manually assign port numbers. I know it's easier to just let it do it itself, and if you're just installing a single development instance of Apps Server, thats fine. But if you're looking at multiple installs per server, (or HA'ing across multiple servers) you really need to know port numbers. This is where staticports.ini comes in handy.

Thankfully OnDemand already had a standard when it came to port numbers, and for the most, I agreed with it. Basically it assigned an environment a number, and then used that number within all ports to do with that environment.

For example, a development environment could be assigned the number 120, webcache listener has the number 45, so the webcache listen port for this development environment is 12045. Another development environment 123 would have the webcache listener as 12345.

Once you get past the initial frustration of things not being where they used to be (ie 7777, 1521 etc etc) it makes complete sense, and you know off the top of your head which port to be going to on whatever environment you're looking at.

OK, so next time I'll talk about install directories, identifying an environment, install users and maybe give you a handy set of scripts to build a website tying it all together... maybe... depends, give me feedback on this first :-)

Also, if you're interested in the list of port numbers across eBusiness Suite, application server (IDM, Portal, J2EE etc) and database, let me know.

Catcha next time.

As you can plainly see, Oracle's new blogging system is up and running. It'll take me a bit to figure this thing out completely, so I apologize in advance for anything that goes awry. (Like I think I just updated all my blog entries, causing the aggregators to go nuts... I know I know, there's only about 10 entries so nobody has probably noticed...)

Anyway, no one's come up with a better complexity scale for me, so it looks like we're set to go... next entry.

If someone wants to be the first to comment on this new blogging system, feel free.

What makes a complex implementation.... no, serious, what does make an environment complex. Is it the number of servers, number of ORACLE_HOMEs, number of different products, maybe size of database, number of databases...

To me, (and I'm not putting my hand up as any kind of expert in this), it's a combination of all of the above.

I reckon there's a few different levels to it though. Let me explain:

If you've just installed eBusiness Suite on a single server, that's not complex. If it's High Availability, ie RAC and Shared Application Tier, then it's kinda complex, (you've probably got at least 4 servers). Add Single SignOn in HA and you're now talking 6 servers, 2 RAC databases and a load balancer... semi complex. Add Portal and Collaboration Suite (both in HA) and now you're talking 14 servers, 4 RAC databases, 11 O_H's... fairly complex. Stick BPEL in there (HA of course) to throw the data around, ContentDB to store the data in, and then dataguard the lot, and you're talking very complex.


But that's just what I think. Anybody have a better/different complexity scale? Anybody Anybody.


So unless anyone has any objections, I'll be talking about the fairly complex to very complex environments. And maybe dabble in the theory of seriously complex.

I realise there's people out there that might stumble across this blog and go "that's not complex, THIS is complex" (to paraphrase Mick Dundee.) If so, please comment when and where neccessary in the next few entries so we can all learn something.


First thing I want to talk about is layout, whether it be hardware, network, SAN, ORACLE_HOMEs, ports etc, it's all pretty much the same. For me it's all about finding the best way to make sure the environment is easy to navigate across, while maintaining fault tolerance and scalability. Once you figure out a pattern for the layout, then it doesn't matter how many servers you have, how many products you're installing, you just need the key and off you go.

But we can talk about that later.

It's been almost a year, and this OCM exam prep is still hanging over my head. Yes I'm still looking at it, but no, I haven't done anything about it...

So, I'm following Sergio's lead... I'm starting over.


I was thinking of topics that might be interesting to both you to read and me to write about, (so I'm more likely to continue posting) and have come up with a few ideas.

Complex Implementations: It's something I'm getting a bit of practice with, and I'm not talking about just a RAC database and a few apps servers here.. one of my last implementations included over 30 ORACLE_HOMEs across 8 products.

The other idea is "YAAB", Yet Another AppsDBA Blog: I'm an AppsDBA by trade, and have worked on all versions from 10.7NCA to R12. But who'd want to read my blog about stuff that people like Stephen Chan wrote about 3 months ago!

I'll probably do a bit of both, but I've got some good ideas for the complex implementations stuff that might go ok. It'll at least run for about 5 or 6 entries, so that should do me till the end of 2009... no, just kidding, I hope to update regularly. (More than once a month would be a good target..)

As well as any updates on my OCP/OCM preparation when/if it happens of course.

What do you think, sound good?

Good. here we go then.

So, 11g Database is out. Plenty of people have blogged about it, so I thought I'd do something a little different. I know I know, you reaallyy want to hear about my lame attempts at studying.. I'll get there, this is much more fun though...

I thought I'd go through an exercise of the whole 11g on VMWare thing, but instead of doing a half baked job and just installing the database, I'll install the lot:

• Oracle Enterprise Linux 5
• 11g CRS
• 11g ASM
• 11g DB

To do that, I need a shared filesystem, which is a little funky in VMWare, so I thought I'd go down the iSCSI route.

Yes, it's been done before, like here and here. But this'll be different, I promise.

I used Starwind from RocketDivision, it's pretty cool and they've got a 30day trial. It was easy to set up, I didn't read any doco on it before the attempt and it's working so I must have done something right...

I've got 2GB memory on my laptop so instead of having 3 VM's, 2 for RAC and one for NAS, and have it GO REALLY SLOW, I can now use my host OS (Windows) to serve out the iSCSI targets, and give 900MB each to the vm's (Windows can run on 200MB memory can't it :-)

After the install, it was just a matter of creating a few "Image File" devices, configuring each with "Asynchronous mode" and "allow multiple concurrent iSCSI connections". Two were created at 300MB for the ocr and vote, and two at 4G for the ASM disks.

So, all that done, lets get to the OS.

I spent a while trying to get Enterprise Linux installer to recognise the scsi disk defined in the vmware, but eventually gave up and used ide. I also spent a while trying to get the installer to pick up an iscsi disk as the root partition, but that didn't work either, maybe I'll try that next... If anyone has created a vm using an iscsi disk to cluster the root filesystem , let me know!

Only one vm was created, so I could get all the rpms installed then clone it.

You've probably all seen the pre-install steps for 10g/9i etc, not much has changed, so I'm not going to put down all the kernel parameters I had to change, and the mindnumbing versions of gcc I needed.. If you want that, why are you reading a blog, go to the source...
Don't you think the new document navigation of the 11g documentation is cool. I like it. Remember the 8i Java stuff that you ended up disabling... HA!

Anyway. Went through all the preinstall stuff, created oracle user, crs user, oinstall, dba blah blah...

Installed asmlib just by putting in:

# up2date -i oracleasm-support oracleasmlib oracleasm-`uname -r`

I think it was about here I cloned the vm to a second instance, so I could start getting to the nitty gritty RACy goodness.

Which is where I'll leave you... stay tuned for the next exciting episode, where I battle the mighty CRS, overcome the devious ASM and triumphantly conquer the illustrious ORACLE 11g DATABASE!!! You never know, I might even put in a screenshot.

Going off track again. I was reading in the AustralianIT News site about "Laptop Lock Down". (I get annoyed having to follow links in blogs, so to summarise it's about security of confidential data on laptops and mobile PCs/Blackberries...)

Being a Consultant, I have several clients across APAC, who would be kinda annoyed if I told them their technical design for their production eBusiness Suite is in some dodgy person's hands...

So I've started using this encrypted filesystem program TrueCrypt, which allows you to create an encrypted file and mount it as a filesystem when unlocked.

Awesome, it means when I get to a client, I mount the drive and all my stuff is there.

And if the laptop goes missing, I don't have to stress about client data falling into the wrong hands. As far as I know, it'd take a while to break through 256 bit AES encryption algorith with SHA-1 Hashing to boot.

Truecrypt is Open Source too which is cool.

OK, back on topic... OCM stuff.

Let's look at the environment which is used in the OCM exam.

Software Environment

• Red Hat Enterprise Linux AS 3 operating system, Oracle Database 10g 10.1.0.4, and Oracle Enterprise Manager 10g Grid Control 10.1.0.3 are used during the exam.
• Participants can choose to work in either Linux and Oracle CLI or GUI environment when available during the exam.
• Candidates will also have access to a complete, non-searchable Oracle documentation set.
  

Time to blow the dust off the VMWare image I started creating.

Basically, I'm going to create an image which includes RHEL3 installed with no "Oracle-ization" done at all, but with all the rpms and Oracle software required to create the environment described above. With a separately staged Oracle Doco CD with the searching turned off.

Then, after each scenario, I can revert to the original snapshot, and start again.

The only issue is stuff like when RAC get's involved, that should be fun.

Anybody got any ideas for that one...apart from bouncing the nearest clients production RAC environment to run a test :-)

I'm assuming those people still sticking with me and reading my Blog (with it's very sporadic nature of posting) are either full on dedicated DBA's, waiting for triumph or failure at the end of this little experiment of mine, or you put this feed in your blog feeder and every time you see an update from me, kick yourself for not getting rid of this blog from the list last time.. :-)

Either way, you're here and I've got you for a minute or two.

I have a question for you, doesn't matter which of the above reader you are:

When was the last time you challenged yourself?

Whether it was as big a challenge such as change of careers, or small, like taking the stairs instead of the elevator twice a week?

hmmm?

I like challenges, don't know why, they stress me out most of the time. I just did a google search on "challenge yourself", and found a blog post from a guy called Dwipal. His post on challenging yourself kinda explains it for me..

"...Short term challenges create a sense of tension in us, and when they are achieved, we feel happy. They also keep us busy and minds occupied until they are resolved...."


I don't think I would have lasted this long as a DBA if I was the type to accept a way of doing something and not change (how many DBA's do you know like that...).

Which leads me to the title of this blog :-) Another challenge!

There was an email blast from Oracle University this morning about the New Oracle Expert Program. A series of exams that recognize specialist DBA's. They include:

• Oracle Database 10g: Managing Oracle on Linux Certified Expert
• Oracle Database 10g: Real Application Clusters Administrator Certified Expert
• Oracle 11i Workflow Certified Expert
• Oracle 11i System Administrator Certified Expert
  

If you're thinking, "oh yay, another way for Oracle to get money from me..." , then I'm not sure why you're reading my blog.... that's not the point of the exams, or of this blog.

So anyway. Those that are like me, and enjoy a challenge, take a look. Those that don't want to give money to the Man, fine.. don't.


Let me know if you're interested or are planning to go for any of them. I'd like to hear how it goes, how you're going to get ready for it. Stuff like that.

Or, better yet, instead of me droning on about my lack of preparation for exam type stuff, someone write a blog entry for me on going for one of the exams... the trepidation leading up, the euphoria of thinking you know the answers, the nervous sweat you seem to break into as you press the "Finished" button....

OK, So I'm going off my usual topic, but this should be fun.

Being an Oracle Consultant, I get to play with quite a few different technologies.

The latest customer challenge I've been given is to implement Secure Enterprise Search. Cool, no problems, except they have a policy of HA'ing all things production.

I've
given you the link above to SES, try going there and looking for "high
availability" :-), then, google "Oracle Secure Enterprise Search high
availability". You get a link to the SES_implementation document which
pretty much states, if you want it HA, have two installed and double
the crawling...

ooookkk... time to get creative..

If
you look at the install. it's fool proof. A one-pager, which means any
"tweaking" probably won't be supported. I can deal with that.

We'll do the install, then see what can be done around it.


OK. So it's installed. Let's have a look at starting / stopping.

$ searchctl startall
$ searchctl stopall

Easy,
fool proof infact, it'll start / stop the database, listener and search
OC4J all in one hit. Hang on. let's look at that again.

$ searchctl
Copyright (c) 2006 Oracle.  All rights reserved.
Unknown command option
Usage::
  midtier -- searchctl start | stop | restart | status
  midtier & backend -- searchctl startall | stopall | restartall

There's a few more options in Searchctl shell script that aren't shown, namely:

start_backend|stop_backend|restart_backend

Which means there's control of midtier and backend processes independantly. Cool. So how to use that to my advantage.

I've
done the install on db_server1, with the database files on an OCFS2
filesystem, I'll move the datafiles out the way and do the same install
from db_server2 server.

OK. all done, database is called the
same thing, in the same storage area. I'll move back the original
database and run the database only on db_server1 and the midtier
processes on db_server2.

Before doing that, I'll change the db
connect strings in any files I can find on the "middle tier" server
db_server2, otherwise it's going to try locally...

for i in `find $ORACLE_HOME`
do
grep -i db2_server $i
if [ $? -eq 0 ]
then
 echo $i
fi
done
(quick and dirty I know but it works)

It's come up with three config files to look at

$ORACLE_HOME/network/admin/tnsnames.ora
$ORACLE_HOME/oc4j/j2ee/OC4J_SEARCH/config/data-sources.xml
$ORACLE_HOME/search/webapp/config/search.properties

Excellent, change connect strings and off we go...

Errors
logging in, makes sense, I was pretty sure the obsfucation of passwords
would be different between database installs. How bout we copy across
the relevant password information from db_server1.

Copy $ORACLE_HOME/oc4j/j2ee/OC4J_SEARCH/config/system-jazn-data.xml from db_server1 to db_server2.

Try logging in again.
http : / / db_server2.localdomain.com/search/admin/index.jsp

Success.

Problem
is, I want middle tier servers running middle tier applications, and
database servers running databases... Makes sense doesn't it.

If
I do another install, it's going to want to install and configure a
database. I don't want that, I've got my database, which I can start
individually.

To create a "middle tier", why not just tar and
copy the ORACLE_HOME across to middle tiers and fire it up. The
inventory won't know that it exists, but that's fine, the whole lot
isn't supportable anyway ;-)

OK. So I've tar'd up the ORACLE_HOME from db_server1 and unpacked it on the midtier servers (mt_server1 and mt_server2).

Frow what we saw before, to start the mid tier only, just run searchctl start (instead of startall).

Now, that's given me two servers with two addresses.
http : / / mt_server1:7777/search/admin/index.jsp
and
http : / / mt_server2:7777/search/admin/index.jsp

Search seems to be working (ie I can log into the admin screen) from both.

Now to create some kind of HA front to them.

I
just happen to have an F5 Load balancer handy to serve out a virtual
server address, and a couple of J2EE Webcache 10.1.3.1.0 installs on
the same mid-tier servers (handy that! :-)

I spose You could
skip this bit, but I like WebCache, so I'm doing it my way. Although I
don't know how it's going to affect Search... we'll see.

OK, in WebCache Admin

Add a listen port. we'll go for 7776, HTTP protocol.

Next, add the origin server
name:mt_server1 port:7777 ping url:/ protocol:HTTP

Create an Alias
name:search_me.localdomain.com port:80 URL Path Prefix:/search

Add the site definition,
name : mt_server1:7776 app_webserver: mt_server1:7777

Add another one,
name: * port:7776 Exclude Fragments.

Restart webcache.

Do
this to both midtier servers. (if you don't know webcache that well.
the above section may be a bit confusing. I'm not putting screen shots
in, if you got this far, you can figure it out, or add a comment)

Fire up the F5 and plug in the virtual server stuff. I'm NAT'ing the port so people don't have to put in the :7776

First for the pool,
Name: search_pool
Health Monitor: tcp
Allow SNAT : Yes
Allow NAT : Yes
Resources
Load Balancing Method: Least Connections (members)
Address: mt_server1 : 7776
Address: mt_server2 : 7776

Now for the Virtual Server

Destination: search_me.localdomain.com
Service Port: 80
Protocol: TCP
Default Pool: search_pool

OK, all done,

Plug in http : / / search_me.localdomain.com/search/admin/index.jsp and off we go. Time to test. :-)

I've logged in. Added a nice simple search source for web (Application Server default home page).

The
log file errors from within the search admin screen, that's fine, nfs
mount point'll fix that if needed. (I noticed the cache and log
directories are stored within the dbf filesystem. Could be interesting
in future...)

Had a quick look at the log file. It's complaining
about not being able to get to www.oracle.com, which makes sense, this
is a test domain which shouldn't be able to get to it. Let's test to
see if the crawl was successful....



Cool. Looks good.

So
there you go, HA'd the front end of Secure Enterprise Search. Next I
spose is to RAC the backend, but that means mucking around with the
original install, instead of building around it. At the moment, if
things go pear shaped, we can shutdown all the midtiers and webcaches,
and send the virtual address for search_me.localdomain.com straight to
db_server1.

OK, so not exactly, but close.

A few people have been hassling me to update what I'm up to, so let's start from the end of the last blog.

It was a fine December morning...

Went on the RedHat System Administration training, which included the RHCT exam at the end. I'm kinda lucky, I've been working on AIX and Linux for the past 7 years, and quite often have to get my hands dirty with Sysadmin stuff. So the course content wasn't completely new to me.

4 days brushing up on linux "stuff" and a 1 day exam. I won't go into exam details here... you'll have to go through it yourself to find out. But lets just say, out of the 10 people doing the exam, 5 bombed out in the first section, therefore failing the exam, 5 made it through to the second section. Yours truly ended up getting 100% for the first section, and 100% for the second section! sweet.

So now I'm an RHCT. RedHat Certified Technician. Where's my red Fedora!

Then, after 4 weeks of lounging around on holidays, getting through post-Christmas fatness and stuff, I find myself on a plane to Hong Kong, where I have 4 weeks to get a major project up and running for their pilot testing and development environments, just a little bit of work...

3 eBusiness Suites, 2 Portals, 2 OID's, 1 Collab Suite and a partridge in a pear tree later. I land back in Perth.

And here I am. Not sure for how long, but we'll see how it goes.

Back to OCP study. Talking of study, I was just reading back my last blog entry, trying to figure out what I know and don't know (yes, you'd think I'd know that...) and realised my last entry gave the wrong Exam Topics link. I should be doing this one as I'm 8i OCP. Almost a disaster.... :-)