Gavin's Blog

OK, So I'm going off my usual topic, but this should be fun.

Being an Oracle Consultant, I get to play with quite a few different technologies.

The latest customer challenge I've been given is to implement Secure Enterprise Search. Cool, no problems, except they have a policy of HA'ing all things production.

I've
given you the link above to SES, try going there and looking for "high
availability" :-), then, google "Oracle Secure Enterprise Search high
availability". You get a link to the SES_implementation document which
pretty much states, if you want it HA, have two installed and double
the crawling...

ooookkk... time to get creative..

If
you look at the install. it's fool proof. A one-pager, which means any
"tweaking" probably won't be supported. I can deal with that.

We'll do the install, then see what can be done around it.


OK. So it's installed. Let's have a look at starting / stopping.

$ searchctl startall
$ searchctl stopall

Easy,
fool proof infact, it'll start / stop the database, listener and search
OC4J all in one hit. Hang on. let's look at that again.

$ searchctl
Copyright (c) 2006 Oracle.  All rights reserved.
Unknown command option
Usage::
  midtier -- searchctl start | stop | restart | status
  midtier & backend -- searchctl startall | stopall | restartall

There's a few more options in Searchctl shell script that aren't shown, namely:

start_backend|stop_backend|restart_backend

Which means there's control of midtier and backend processes independantly. Cool. So how to use that to my advantage.

I've
done the install on db_server1, with the database files on an OCFS2
filesystem, I'll move the datafiles out the way and do the same install
from db_server2 server.

OK. all done, database is called the
same thing, in the same storage area. I'll move back the original
database and run the database only on db_server1 and the midtier
processes on db_server2.

Before doing that, I'll change the db
connect strings in any files I can find on the "middle tier" server
db_server2, otherwise it's going to try locally...

for i in `find $ORACLE_HOME`
do
grep -i db2_server $i
if [ $? -eq 0 ]
then
 echo $i
fi
done
(quick and dirty I know but it works)

It's come up with three config files to look at

$ORACLE_HOME/network/admin/tnsnames.ora
$ORACLE_HOME/oc4j/j2ee/OC4J_SEARCH/config/data-sources.xml
$ORACLE_HOME/search/webapp/config/search.properties

Excellent, change connect strings and off we go...

Errors
logging in, makes sense, I was pretty sure the obsfucation of passwords
would be different between database installs. How bout we copy across
the relevant password information from db_server1.

Copy $ORACLE_HOME/oc4j/j2ee/OC4J_SEARCH/config/system-jazn-data.xml from db_server1 to db_server2.

Try logging in again.
http : / / db_server2.localdomain.com/search/admin/index.jsp

Success.

Problem
is, I want middle tier servers running middle tier applications, and
database servers running databases... Makes sense doesn't it.

If
I do another install, it's going to want to install and configure a
database. I don't want that, I've got my database, which I can start
individually.

To create a "middle tier", why not just tar and
copy the ORACLE_HOME across to middle tiers and fire it up. The
inventory won't know that it exists, but that's fine, the whole lot
isn't supportable anyway ;-)

OK. So I've tar'd up the ORACLE_HOME from db_server1 and unpacked it on the midtier servers (mt_server1 and mt_server2).

Frow what we saw before, to start the mid tier only, just run searchctl start (instead of startall).

Now, that's given me two servers with two addresses.
http : / / mt_server1:7777/search/admin/index.jsp
and
http : / / mt_server2:7777/search/admin/index.jsp

Search seems to be working (ie I can log into the admin screen) from both.

Now to create some kind of HA front to them.

I
just happen to have an F5 Load balancer handy to serve out a virtual
server address, and a couple of J2EE Webcache 10.1.3.1.0 installs on
the same mid-tier servers (handy that! :-)

I spose You could
skip this bit, but I like WebCache, so I'm doing it my way. Although I
don't know how it's going to affect Search... we'll see.

OK, in WebCache Admin

Add a listen port. we'll go for 7776, HTTP protocol.

Next, add the origin server
name:mt_server1 port:7777 ping url:/ protocol:HTTP

Create an Alias
name:search_me.localdomain.com port:80 URL Path Prefix:/search

Add the site definition,
name : mt_server1:7776 app_webserver: mt_server1:7777

Add another one,
name: * port:7776 Exclude Fragments.

Restart webcache.

Do
this to both midtier servers. (if you don't know webcache that well.
the above section may be a bit confusing. I'm not putting screen shots
in, if you got this far, you can figure it out, or add a comment)

Fire up the F5 and plug in the virtual server stuff. I'm NAT'ing the port so people don't have to put in the :7776

First for the pool,
Name: search_pool
Health Monitor: tcp
Allow SNAT : Yes
Allow NAT : Yes
Resources
Load Balancing Method: Least Connections (members)
Address: mt_server1 : 7776
Address: mt_server2 : 7776

Now for the Virtual Server

Destination: search_me.localdomain.com
Service Port: 80
Protocol: TCP
Default Pool: search_pool

OK, all done,

Plug in http : / / search_me.localdomain.com/search/admin/index.jsp and off we go. Time to test. :-)

I've logged in. Added a nice simple search source for web (Application Server default home page).

The
log file errors from within the search admin screen, that's fine, nfs
mount point'll fix that if needed. (I noticed the cache and log
directories are stored within the dbf filesystem. Could be interesting
in future...)

Had a quick look at the log file. It's complaining
about not being able to get to www.oracle.com, which makes sense, this
is a test domain which shouldn't be able to get to it. Let's test to
see if the crawl was successful....



Cool. Looks good.

So
there you go, HA'd the front end of Secure Enterprise Search. Next I
spose is to RAC the backend, but that means mucking around with the
original install, instead of building around it. At the moment, if
things go pear shaped, we can shutdown all the midtiers and webcaches,
and send the virtual address for search_me.localdomain.com straight to
db_server1.

So, 11g Database is out. Plenty of people have blogged about it, so I thought I'd do something a little different. I know I know, you reaallyy want to hear about my lame attempts at studying.. I'll get there, this is much more fun though...

I thought I'd go through an exercise of the whole 11g on VMWare thing, but instead of doing a half baked job and just installing the database, I'll install the lot:

• Oracle Enterprise Linux 5
• 11g CRS
• 11g ASM
• 11g DB

To do that, I need a shared filesystem, which is a little funky in VMWare, so I thought I'd go down the iSCSI route.

Yes, it's been done before, like here and here. But this'll be different, I promise.

I used Starwind from RocketDivision, it's pretty cool and they've got a 30day trial. It was easy to set up, I didn't read any doco on it before the attempt and it's working so I must have done something right...

I've got 2GB memory on my laptop so instead of having 3 VM's, 2 for RAC and one for NAS, and have it GO REALLY SLOW, I can now use my host OS (Windows) to serve out the iSCSI targets, and give 900MB each to the vm's (Windows can run on 200MB memory can't it :-)

After the install, it was just a matter of creating a few "Image File" devices, configuring each with "Asynchronous mode" and "allow multiple concurrent iSCSI connections". Two were created at 300MB for the ocr and vote, and two at 4G for the ASM disks.

So, all that done, lets get to the OS.

I spent a while trying to get Enterprise Linux installer to recognise the scsi disk defined in the vmware, but eventually gave up and used ide. I also spent a while trying to get the installer to pick up an iscsi disk as the root partition, but that didn't work either, maybe I'll try that next... If anyone has created a vm using an iscsi disk to cluster the root filesystem , let me know!

Only one vm was created, so I could get all the rpms installed then clone it.

You've probably all seen the pre-install steps for 10g/9i etc, not much has changed, so I'm not going to put down all the kernel parameters I had to change, and the mindnumbing versions of gcc I needed.. If you want that, why are you reading a blog, go to the source...
Don't you think the new document navigation of the 11g documentation is cool. I like it. Remember the 8i Java stuff that you ended up disabling... HA!

Anyway. Went through all the preinstall stuff, created oracle user, crs user, oinstall, dba blah blah...

Installed asmlib just by putting in:

# up2date -i oracleasm-support oracleasmlib oracleasm-`uname -r`

I think it was about here I cloned the vm to a second instance, so I could start getting to the nitty gritty RACy goodness.

Which is where I'll leave you... stay tuned for the next exciting episode, where I battle the mighty CRS, overcome the devious ASM and triumphantly conquer the illustrious ORACLE 11g DATABASE!!! You never know, I might even put in a screenshot.