Thursday Jan 31, 2013

Deep Inside Every Sysadmin Is ...

#WWI-Ace

photo courtesy of James Vaughan - all rights reserved

... an Oracle ACE!

The thrills. The glory. The fame. Who can resist?

Turns out sysadmins can.

Last time I checked, the Oracle ACE program had 417 ACES and ACE Directors. Of those 417, only 6 have a specialty in Oracle Solaris or Oracle Linux.

That's simply not enough to defend the homeland! I know there are many more sysadmins and developers proficient in Oracle Solaris and Oracle Linux who can qualify to become Oracle ACES. Toss your silk scarf around your neck and grab your goggles. Then talk to me on the OTN Garage page on Facebook. I'll explain the benefits and help you enroll in pilot training.

Current Oracle ACES With Specialties in Oracle Solaris or Oracle Linux

Suk Kim, ACE Director, Oracle Solaris, Korea

Proficient in Oracle Solaris system tuning, troubleshooting Oracle Solaris security, audit information security, penetration tester incident and response, digital forensics virtualization, and cloud computing. Member of Korea Oracle Solaris User Network, Chairman of Oracle Solaris Tehchnet, Manager of Solaris School, adjunct professor at Ansan University, senior consultant at NoBreak Co., LTD.

Diego Aguirre, ACE, Oracle Solaris, Argentina

Diego Aguirre has been a Solaris Support Specialist since 1998. Over the past several years, he has contributed to the Oracle Solaris Community and has published technical articles for Sun Microsystems and now Oracle. He is the author of http://solaris4ever.blogspot.com.

Alexander Eremin, ACE, Oracle Solaris, Russia

Alexander Eremin is a user on Solaris and Linux platforms since 1995. Over the past ten years, he has worked as a Senior Unix Administrator. He is also the creator of the MilaX - Small Live Distribution of OpenSolaris. Alexander is also taking part in the Caiman OpenSolaris project.

Julien Gabel, ACE, Oracle Solaris, France

Julien Gabel is a Multi-platform UNIX systems consultant and administrator in mutualized and virtualized environments. He has architecture and expertise in building Solaris and UNIX experience in large enterprises such as banking and financial services, IT services, Telecoms and multimedia companies.

Raimonds Simanovskis, ACE, Oracle Linux, Latvia

Raimonds Simanovskis in founder of EazyOne which develops business intelligence web application eazyBI.com. Previously he was working at Tieto Latvia where he was using and promoting new technologies, open source and Agile software development. Raimonds has participated in many Oracle E-Business Suite implementation projects as well as Oracle based software development projects. In recent years he is active Ruby language and Ruby on Rails framework user and contributor. He has created and maintains Oracle database adapter for Ruby on Rails as well as PL/SQL and Ruby integration libraries.

Damian Wojslaw, ACE, Oracle Solaris, Poland

Damian is currently working as systems operator since 1999. Since 2006 he has worked with Solaris and OpenSolaris operating systems and other Sun Microsystems born applications. He blogs regularly on TrochejEN and reposts on Planet OpenSolaris. Damian has translated four OpenSolaris related Guides (ZFS Administrator Guide, OpenSolaris Installation Guide: Basic Installations, DTrace User Guide, Device Driver Tutorial) to Polish.

Defend the homeland!

- Rick

Website Newsletter Facebook Twitter My Personal Blog

Thursday Jan 10, 2013

Recent How-To Articles About Oracle Solaris Zones

LEGO Clone Army Collection

How to Put Oracle Solaris Zones on Shared Storage for Easy Cloning

by Jeff Victor

What is ZOSS? Zones on shared storage. Why would you do that? When you configure a zone on shared storage, you can quickly clone it on any server that uses that storage. Jeff explains how.

How to Create Oracle Solaris 11 Zones with Oracle Enterprise Manager Ops Center

You can also create Oracle Solaris 11 zones with Oracle Enterprise Manager Ops Center 12c. When you set up a zone this way, you can add the zone to a server pool and use the zone migration feature of Oracle Solaris 11. In this article, Laura Hartman shows you how to create and configure a non-global zone from the Plan Management View of this handy-dandy tool.

How to Create a Load-Balanced Application Across Two Oracle Solaris Zones

by Laura Hartman

Install Apache Tomcat on two Oracle Solaris zones. Connect them across a VPN. And let the Integrated Load Balancer in Oracle Solaris 11 manage traffic. Presto: high(er) availability in a single server.

- Rick

Website Newsletter Facebook Twitter My Personal Blog

Tuesday Jan 08, 2013

How to Upgrade an Oracle Solaris 11 Repository with 11.1 Packages

These instructions assume you already have a local Oracle Solaris 11 11/11 IPS package repository, and you want to update that local repository by adding Oracle Solaris 11.1 packages.

You can either use the pkgrecv command or you can download a repository image file.

Using pkgrecv

Using pkgrecv takes more time and requires your system to be connected to the Internet for a much longer time. If you use pkgrecv, specify http://pkg.oracle.com/solaris/release as the source, and be sure to specify the -m all-versions option. See the pkgrecv(1) man page for more information.

Using a Repository Image File

If you prefer to use a repository image file, first download the image file and then copy the contents to your existing repository. You can get the Oracle Solaris 11.1 repository image files from OTN or from eDelivery. Then follow these instructions.

Step 0. Assume the root role and snapshot your local Oracle Solaris 11 11/11 repository.

In this example, your local Oracle Solaris 11 11/11 repository is located at /export/repo/Solaris11.

# zfs snapshot rpool/export/repo/Solaris11@11-1111

Step 1. Download Oracle Solaris 11 from OTN.

Go to the download page. Read the "OTN License Agreement for Oracle Solaris" and click the Accept License Agreement button.

Step 2. Click the "MD5 checksum" link to display the checksums for the files you want to download.

Step 3. Scroll down to "Oracle Solaris 11.1 Repository Image."

Step 4. Click "Download Part A SPARC, x86" to get the file sol-11_1-repo-full.iso-a.

Verify the checksum like this:

# digest -a md5 sol-11_1-repo-full.iso-a

Step 5. Click "Download Part B SPARC, x86" to get the file sol-11_1-repo-full.iso-b.

Verify the checksum.

# digest -a md5 sol-11_1-repo-full.iso-b

Step 6. Click "Download Oracle Solaris 11 Pre-Upgrade Repository Image SPARC, x86 to get the file sol-11_1-upgrade-repo.iso.

Verify the checksum.

# digest -a md5 sol-11_1-upgrade-repo.iso

Step 7. Create one image file from the Part A and Part B files.

Verify the checksum of the resulting image file.

# cat sol-11_1-repo-full.iso-a sol-11_1-repo-full.iso-b > sol-11_1-repo-full.iso

Step 8. Merge the Oracle Solaris 11.1 packages from the repository image file into your local Oracle Solaris 11 11/11 repository.

# mount -F hsfs sol-11_1-repo-full.iso /mnt
# rsync -aP /mnt/repo/ /export/repo/Solaris11
# umount /mnt

Step 9. Merge packages from the Oracle Solaris 11 pre-upgrade repository image into your local Oracle Solaris 11 11/11 repository.

# mount -F hsfs sol-11_1-upgrade-repo.iso /mnt
# rsync -aP /mnt/repo/ /export/repo/Solaris11
# umount /mnt

Step 10. Catalog new packages.

# pkgrepo refresh -s /export/repo/Solaris11
# pkg refresh solaris

You can use the pkgrepo info and pkgrepo get commands to check the properties set on the updated repository.

Step 11. Snapshot your updated repository.

# zfs snapshot rpool/export/repo/Solaris11@11.1

Step 12. Check that your Solaris publisher origin is set to your local repository.

- Alta Elstad

Website Newsletter Facebook Twitter

Thursday Jan 03, 2013

Extreme Sports and the SPARC T5 Chip

In my day, you were extreme if you surfed. Nobody had a leash. If a wave ripped the board out of your hands, you had to swim all the way back to the beach. In big surf, that could take 15 minutes. And then you had to paddle out again.

Today, if you're not juggling rusty chainsaws while riding a BMX bike off the top of the Eiffel Tower with half a dozen angry chimpanzees trying to rip off your helmet and goggles, you're not considered extreme.

Which is exactly why the SPARC T5 chip has 1024 functional CPUs. None of Oracle's SPARC engineers wanted to find himself at a cocktail party having to confess in an embarrassed chortle to a salon full of top hats and sequins that he had designed a mere 512 CPU chip. Imagine the chagrin!

Interview: Deep Dive into the SPARC T5 Chip

So Oracle's SPARC engineers worked wicked hard to scale the T5 to eight sockets. Since each socket has 16 cores, that gives you 128 cores, total. Since each core can support eight individual threads (or strands, if you're not the sartorial type), you wind up with a total of 1024 functional CPUs.

As you know, processing power without bandwidth is kinda like a mega motor with a nano fuel tank. Doesn't get you too far. So the T5 also has memory bandwidth to match its processing power. And lots of other capabilities that you an read about in this:

Interview: Deep Dive into the SPARC T5 Chip

- Rick

P.S., If you want to read about surfing in the old days, check out Chapter 1 of Tocayos, a novel I've been writing and posting online in my spare time.
Website Newsletter Facebook Twitter My Personal Blog

Thursday Dec 20, 2012

Top 10 Articles of 2012 Include Oracle Solaris, Linux, Virtualization

source

That's a 72 Norton Commando fashioned into a cafe racer. Heavy.com named a newer version the #1 bike in the 2012 New York International Motorcycle Show. (I didn't like Heavy.com's picture, so I found a better one from the blog listed as source, above.)

OTN also has an annual top 10. In that post by Bob Rhubart, from OTN's Architect community, six of the top ten technical articles were about technologies of interest to system admins and developers.

Boo-yah!

#2 - How Dell Migrated from SUSE Linux to Oracle Linux

by Jon Senger, Aik Zu Shyong, and Suzanne Zorn

In June of 2010, Dell made the decision to migrate 1,700 systems from SUSE Linux to Oracle Linux, while leaving the hardware and application layers unchanged. The people who worked on the migration describe how Dell planned and implemented the migration, including key conversion issues and an overview of their transition process.

#4 - Getting Started with Oracle Unbreakable Enterprise Kernel Release 2

by Lenz Grimmer

How to update your Oracle Linux systems to the latest version of the Unbreakable Enterprise Kernel. Switching is easy—applications and the operating system remain unchanged. There is no need to perform a full re-install; only the relevant RPM packages are replaced.

#6 - How to Use Oracle VM VirtualBox Templates

by Yuli Vasiliev

This article explains how to use Oracle VM VirtualBox Templates in Oracle VM VirtualBox. It is similar to the article that explains how to prepare an Oracle VM environment to use Oracle VM Templates, but it describes how to download, install, and configure the templates within Oracle VM VirtualBox, instead of on bare metal.

#7 - How to Update Oracle Solaris 11 Systems From Oracle Support Repositories

by Glynn Foster

You may already know that you don't have to worry about manually tracking and validating patch dependencies when you update a version of Oracle Solaris 11. This makes updates much easier. Glynn Foster demonstrates how easy it is to update the OS from a support repository, and how to make sure everything went well.

#8 - Tips for Hardening an Oracle Linux Server

by Lenz Grimmer and James Morris

General strategies for hardening an Oracle Linux server. Oracle Linux comes "secure by default," but the actions you take when deploying the server can increase or decrease its security. How to minimize active services, lock down network services, and many other tips.

#9 - How to Create a Local Yum Repository for Oracle Linux

by Jared Greenwald

How to create a local yum repository for Oracle Linux, and configure up2date and yum to install and update packages from the repositories.

More About OTN's Technical Articles

See all system admin- and systems developer-related technical articles published on OTN here.

Interested in publishing an article on OTN? Click here or join the conversation on the OTN Garage Facebook page.

- Rick

Website Newsletter Facebook Twitter

Monday Dec 17, 2012

The Loneliest Road in America and the OTN Garage

Source

I never told anyone how the image of the OTN Garage on Facebook came to be.

I took the Facebook picture on Route 50 in Nevada, USA, in October of 2010. I was riding from Colorado to Oracle OpenWorld in San Francisco, so it was probably October. Route 50 is known as "The Loneliest Road in America." There are roads across Nevada that have even LESS traffic, but Route 50 still one. desolate. road.

Although I have seen stranger things while riding along Nevada's Extraterrestrial Highway, I still run across notable oddities every time I ride Route 50. Like the old man with a bandolero of water bottles jogging along the side of the highway in the middle of the day, 50 miles from the closest town. First ultra-marathoner I'd seen in action. He waved at me. Or the dozen Corvettes with California license plates driving toward me, all doing the speed limit in the middle of nowhere because they were being tailed by half a dozen Nevada state troopers. #fail.

I don't remember which town I was in, but I noticed the building when I stopped at the gas station. While standing there pouring fuel into the Harley, the store caught my eye. So I pulled the bike in front and walked inside. The owner is a little old lady, about 100 years old. Most of the goods she had on the shelves looked like they had been placed there during WWII. She was itty bitty and could barely see over the counter, but she was so happy when I bought a bar of Hershey's chocolate that she gave me a five cent discount.

I took a few pictures and, when I got back, Kemer Thomson, who sometimes blogs here, photoshopped the OTN Garage and Oil Change signs onto it.

The bike is a 2009 Road King Classic with a Bob Dron fairing and a Corbin heated seat. The seat came in handy when I rode home over Tioga Pass. The Road King is a very comfy touring bike with a great Harley rumble. I'm kinda sorry I sold it.

When I stopped for fuel about 75 miles down the road at the next town, I peeled back the chocolate bar. It had turned into powder. Probably 50 years ago.

- Rick

Website Newsletter Facebook Twitter

Tuesday Nov 13, 2012

We Need More Migration!

source

Eva Mendez says, "Oye chico, do you really want to keep your data in that tired legacy file system when it could be enjoying encryption, compression, deduplication, snapshots, remote replication and other benefits provided by ZFS in Oracle Solaris 11?

It's really not that hard to cross over. If you know how."

"I don't know how, me dices? Esta bien, papacito. Go to OTN. Take my word for it. They know how."

<blushing>
Aw shucks, Eva. Anything for you!
</blushing>

The Best Way to Migrate Data From Legacy File Systems to ZFS

To migrate data from a legacy filesystem to ZFS in Oracle Solaris 11, you need to install the shadow-migration package and enable the shadowd service. Then follow the simple procedure described by Dominic Kay.

How to Update to Oracle Solaris 11 Using the Image Packaging System

Oracle Solaris 11.1 has been released. You can upgrade using either Oracle's official Solaris release repository or, if you have a support contract, the Support repository. Peter Dennis explains how.

How to Migrate Oracle Database from Oracle Solaris 8 to Oracle Solaris 11

How to use the Oracle Solaris 8 P2V (physical to virtual) Archiver tool, which comes with Oracle Solaris Legacy Containers, to migrate a physical Oracle Solaris 8 system with Oracle Database and an Oracle Automatic Storage Management file system into an Oracle Solaris 8 branded zone inside an Oracle Solaris 10 guest domain on top of an Oracle Solaris 11 control domain.

- Ricardo

Website Newsletter Facebook Twitter

Monday Oct 29, 2012

Is This Your Idea of Disaster Recovery?

Don't just make do with less.

Protect what you've got.

By, for instance, deploying Oracle Solaris 10 inside a zone cluster.

"Wait," you say, "what is a zone cluster?"

It is a zone deployed across different physical servers.

"Who would do that!" you ask in a mild panic.

Why, an upstanding sysadmin citizen interested in protecting his or her employer's investment with appropriate high availability and disaster recovery. If one server gets wiped out by Hurricane Sandy along with pretty much the entire East Coast of the USA, your zone continues to run on the other server(s). Provided you set them up in Edinburgh. This white paper (pdf) explains what a zone cluster is and how to use it. If a white paper reminds you of having to read War and Peace in school, just use this Oracle RAC and Solaris Cluster Cheat Sheet, instead.

"But wait!" you exclaim. "I didn't realize Solaris 10 offered zone clusters!"

I didn't, either! And in an earlier version of this blog post I said that zone clusters were only available with Oracle Solaris 11. But Karoly Vegh pointed me to the documentation for Oracle Solaris Cluster 3.3, which explains how to manage zone clusters in Oracle Solaris 10. Bite my fist!

So, the point I was trying to make is not just that you can run Oracle Solaris 10 zone clusters, but that you can run them in an Oracle Solaris 11 environment. Now let's return to our conversation and pick up where we left off ...

"Oh no! Whatever shall I do?"

Fear not. Remember how Oracle Solaris 11 lets you create a Solaris 10 branded zone inside a system running Oracle Solaris 11? Well, the Solaris Cluster engineers thought that was a bang-up idea, and decided to extend Oracle Solaris Cluster so that you could run your Solaris 10 applications inside the protective cocoon of an Oracle Solaris 11 zone cluster. Take advantage of the installation improvements and network virtualization capabilities of Oracle Solaris 11 while still running your application on Oracle Solaris 10. You Luddite, you.

That capability is in the latest release of Oracle Solaris Cluster, version 4.1, which became available last Friday.

"Last Friday! Is it too late to get a copy?"

You can still get a free copy from our download center (see below). And, if you'd like to know what other goodies the 4.1 release of Oracle Solaris Cluster provides, see:

As always, you can get the latest information about Oracle Solaris Cluster, plus technical how-to articles, documentation, and more from Oracle Solaris Cluster Resource Page for Sysadmins and Developers.

And don't forget about the online launch of Oracle Solaris 11.1 and Oracle Solaris Cluster 4.1, scheduled for Nov 7.

"I feel so much better, now!"

Think nothing of it. That's what we're here for.

- Rick
Website Newsletter Facebook Twitter

Thursday Oct 25, 2012

Yay! Oracle Solaris 11.1 Is Here!

Even the critters are happy.

This is no cosmetic release. It's got TONS of new stuff for both system admins and system developers. In the coming weeks and months I'll highlight specific new capabilities, but for now, here are a few resources to get you started.

What's New (pdf)

Describes enhancements for sysadmins in:

  • Installation
  • System configuration
  • Virtualization
  • Security and Compliance
  • Networking
  • Data management
  • Kernel/platform support
  • Network drivers
  • User environment

And for system developers:

Download

Free downloads for SPARC and x86 are available, along with instructions and tips for using the new repositories and Image Packaging System.

Tech Article: How to Upgrade to Oracle Solaris 11.1

You can upgrade using either Oracle's official Solaris release repository or, if you have a support contract, the Support repository. Peter Dennis explains how.

Documentation

Superbly written instructions from our dedicated cadre of world-renowned but woefully underpaid technical writers:

  • Getting Started
  • Installing, Booting, and Updating
  • Establishing an Oracle Solaris Network
  • Administering Essential Features
  • Administering Network Services
  • Securing the Operating System
  • Monitoring and Tuning
  • Creating and Using Virtual Environments
  • Working with the Desktop
  • Developing Applications
  • Reference Manuals
  • And more

Training

And don't forget the new online training courses from Oracle University! I really liked them. Here are my first and second impressions. - Rick

Website Newsletter Facebook Twitter

Friday Oct 05, 2012

Hurry! See the uncensored OOW videos before they get edited!

source

Uploaded so far:

Which Oracle Solaris 11 Technologies Have Sysadmins Been Using Most?

Director's Cut - Uncensored - Markus Flierl, VP Solaris Core Engineering, describes how Oracle Solaris 11 customers are taking advantage of the Image Packaging System and the snapshot capability of ZFS to run more frequent updates of not only the OS, but also the applications (agile development, anyone?), and how they're using the network virtualization capabilities in Oracle Solaris 11 to isolate applications and manage workloads on the cloud.

Watch How Hybrid Columnar Compression Saves Storage Space

Director's Cut - Uncensored - Art Licht shows how hyprid columnar compression (HCC) compresses data 30x without slowing down other queries that the database is performing. First he shows what happens when he runs database queries without HCC, then he shows what happens when he runs the queries with HCC.

Security Capabilities and Design in Oracle Solaris 11

Director's Cut - Uncensored - Compliance reporting. Extended policy. Immutable zones. Three of the best minds in Oracle Solaris security explain what they are, what customers are doing with them, and how they were engineered. Filmed at Oracle Open World 2012.

Why DTrace and Ksplice Have Made Oracle Linux 6 Popular with Sysadmins

Use the DTrace scripts you wrote for Oracle Solaris on Oracle Linux without modification. Wim Coekaerts, VP of Engineering for Oracle Linux, explains how this capability of DTrace, the zero downtime updates enabled by KSplice, and other performance and stability enhancements have made Oracle Linux 6 popular with sysadmins.

Why Solaris 11 Is Being Adopted Faster Than Solaris 10

Sneak Preview - Uncut Version - Lynn Rohrer, Director of Oracle Solaris Product Management explains why customers are adopting Oracle Solaris 11 at a faster rate than Oracle Solaris 10, and proves why you should never challenge a Montana woman to a test of strength.

What Forsythe Corp Is Helping Its Customers Do With Oracle Solaris 11

Director's Cut - Unedited - Lee Diamante, Solutions Architect for Forsythe Corp, an Oracle Solaris Partner, explains why Forsythe has been recommending Oracle Solaris to its customers, and what those customers have been doing with it.

Lots more to come ...

- Rick

Website Newsletter Facebook Twitter

Thursday Sep 27, 2012

Heading Out to Oracle Open World

In case you haven't figured it out by now, Oracle reserves an awful lot of announcements for Oracle Open World. As a result, the show is always a lot of fun for geeks. What will the Oracle Solaris team have to say? Will the Oracle Linux team have any surprises? And what about Oracle hardware?

For my part, I'll be one of the lizards at the OTN Lounge with the OTN crew, handing out t-shirts to system admins and developers, or anyone who is willing to impersonate one. I understand, not everyone can have the raw animal magnetism of a sysadmin, or the debonair sophistication of a C++ developer, so some of you have no choice but to pretend. I won't judge.

I'll also be doing video interviews of as many techie people as I can corner. I've got more than 30 interviews already scheduled. Most of them will be 3-5 minutes long. I'll be asking our best technical minds what's cool about their latest technologies and what impact it will have on system admins or system developers. I'll be posting those videos here:

Find OTN Systems Videos from Oracle Open World Here!

We've got some great topics in mind. A dummies guide to hardware-assisted cryptography with Glenn Brunette. ZFS deduplication. The momentum building around Oracle Solaris 11, with Lynn Rohrer, plus conversations with partners who have deployed Oracle Solaris 11. Migrating to Oracle Database with SQL Developer. The whole database cloud thing. Oracle VM and, of course, Oracle Linux.

So even if you can't be part of the fun, keep an eye out for the videos on our YouTube channel.

- Rick

Website Newsletter Facebook Twitter

Friday Sep 14, 2012

New Hands-On Labs For Oracle VM

I just spent some time walking through the labs that Christophe Pauliat and Olivier Canonge prepared to help you become familiar with Oracle VM. They are terrific. We will offer them for the first time at Oracle Open World. Because they require some pre-work and 16Gigs of memory, we are supplying the laptops for the participants.

Lab 1: Deploying Infrastructure as a Service with Oracle VM

Session ID: HOL9558
Tuesday October 2nd, 2012
10:15am – 11:15am
Marriott Marquis - Salon 14/15

Planning and deployment of an infrastructure as a service (IaaS) environment with Oracle VM as the foundation. Storage capacity planning, LUN creation, network bandwidth planning, and best practices for designing and streamlining the environment so that it's easy to manage.

Lab 2: Virtualize and Deploy Oracle Applications Using Oracle VM Templates

Session ID: HOL9559
Tuesday October 2nd, 2012
11:45am – 12:45pm
Marriott Marquis - Salon 14/15

How to deploy Oracle applications in minutes with Oracle VM Templates. Step-by-step lab proctored by field-experienced engineers and product experts. Covers:

  • Find out what Oracle VM Templates are and how they work
  • Deploy an actual Oracle VM Template for an Oracle Application
  • Plan your deployment to streamline on going updates and upgrades

Lab 3: x86 Enterprise Cloud Infrastructure with Oracle VM 3.x and Sun ZFS Storage Appliance

Session ID: HOL 9870
Wednesday, 3 Oct, 2012
5:00 PM - 6:00 PM
Marriott Marquis - Salon 14/15

This hands-on lab will demonstrate what Oracle’s enterprise cloud infrastructure for x86 can do, and how it works with Oracle VM 3.x. It covers:

  • How to create VMs
  • How to migrate VMs
  • How to deploy Oracle applications quickly and easily with Oracle VM Templates
  • How to use the Storage Connect plug-in for the Sun ZFS Storage Appliance

Additional Virtualization Resources for Sysadmins

- Rick

Website Newsletter Facebook Twitter

Wednesday Sep 12, 2012

Reaping the Benefits of the Image Packaging System

source

One of the promises made about Oracle Solaris 11 was easier installation. Remember?

Do you also remember how involved installing Oracle Solaris Cluster used to be? It was so involved, in fact, that we (when we were Sun Microsystems) wouldn't even let you do it yourself.

How times have changed.

New - How to Automate The Installation of Oracle Solaris Cluster 4.0

Thanks to the new image packaging architecture in Oracle Solaris 11, you can now automate the installation of Oracle Solaris Cluster 4.0. Why is that such a big deal? As Lucia Lai explains it:

"Without the AI, you would have to manually install the cluster components on the cluster nodes, and then run the scinstall tool to add the nodes to the cluster. If, instead, you use the AI, both the Oracle Solaris 11 and the Oracle Solaris Cluster 4.0 packages are installed onto the cluster nodes directly from Image Packaging System (IPS) repositories, and the nodes are booted into a new cluster with minimum user intervention."

Lucia goes on to explain how to set up and configure the AI server, how to plan your cluster configuration for the automated installation, how to use the scinstall utility, how to set up the DHCP server, and more. A thorough, well-written article.

- Rick

Website Newsletter Facebook Twitter

Tuesday Aug 21, 2012

Worth the Money?

source

Learning a new technology really is the modern equivalent of doing the Ulysees thing in Homer's Odyssey. If you're the person who has to actually deploy the darned thing. And keep it running.

First, you have to wade through the marketing designed to mesmerize your boss ...

The eData Cloud-Optimized Storage Environment solution increases the adaptability of scalable business continuity while protecting infrastructure integrity optimized for the demands of reliability, availability, and security expressly designed for the unique requirements of the data center while enhanced for today's particular, unique, and demanding enterprise challenges. In a heterogenous computing environment.

So you shake your head vigorously in the hope that most of those words will fall out your ears, and go to the documentation, which is wicked, wicked useful. Once once you have a good idea of what you want to do. But frustrating as hell when you're not sure what you're supposed to be doing. Or why.

The technical articles that OTN publishes help a lot, but they don't give you the complete picture, do they? You wind up knowing how to do some really cool things, but not having a clue how to do others. Or worse: not knowing if there are other things you need to know.

So you go to the forums. And ask a question. OTN's forums are pretty good, but even in our forums you might not get an answer. And you might develop a lasting relationship with somebody born in San Quentin Prison who dedicates himself to stalking you for the rest of his life for wasting 18 seconds of his precious time.

We're all used to this, and repeat it hundreds of times throughout the year.

But wouldn't it be nice to learn something the easy way? Just once? Have somebody who really knows what they're talking about give us the complete picture? First at the high level so we get to see all the pieces and finally understand what it is we're dealing with. That alone is almost priceless. But also in full detail, so we know how to actually install, deploy, manage, and update a technology. From end to end. Because we've done it ourselves. More than once.

For me, that would be Christmas in August. The catch for most sysadmins nowadays is that there just isn't enough time to take a class. You can't get away from the office long enough without the place burning down. Which is why Oracle University came up with its on-demand format. Here's one example:

On Demand Training: Transition to Oracle Solaris 11

Like the average sysadmin, I have little to no free time during my work week. So I can't sign up for a week-long class. And even if I did, I wouldn't pay attention half the time because I'd be answering emails, IM's, and phone calls. So this on-demand format really works for me. Plus, the content is really good. An example of how the instructor sets the context for the new installation tools in Oracle Solaris 11, with just a few words:

"Now, speaking of Solaris installations, we have essentially three different ways that we can install this. We have the automated installer. Now, the automated installer is the replacement for JumpStart. The idea here is we're installing across the network. We have a manifest that lists what component should get installed. We have client profiles that say OK, these are the clients that should get the software.

"Then we have a couple of different interactive installation options. We have a LiveCD. Now, LiveCD is designed for the desktop environment. It has a GUI environment. So for those of you that are dealing with installations that are going to happen on a desktop or notebook computers, generally, you're going to do a LiveCD installation of that. Then we have the text installer. That's typically what you're probably used to in server deployments where it's a text-based interface where you're answering the questions to install the operating system so that you're not having to worry about the resources of a graphical environment."

If you're wondering why I'm blogging about this course on OTN Garage (again), it's simple: I'm taking the course right now, in between my other work, and I'm freakin' loving it! In my case, Oracle is paying for it. But after decades of trying to learn this technology on my own --with access to Oracle's engineers, mind you-- even if Oracle didn't pay for it, I'd be awfully tempted to stop buying motorcycles and pay for it myself. Just for the peace of mind. For the relief of being certain that I know what I'm talking about.

If the link above doesn't work for you, try this one.

- Rick

Website Newsletter Facebook Twitter

Friday Aug 17, 2012

How to Create More Oracle Solaris 11 Zones With Less Effort

If you are familiar with zones in Oracle Solaris 11, you already know how to create them using a procedure like the one described in this article:

How to Get Started Creating Zones in Oracle Solaris 11
Duncan Hardie demonstrates how to perform basic operations with zones: first, how create a single zone using the command line, then how to add an application to a zone, and finally how to clone a zone.

And you may be aware that you can configure your zones so that they are easier to clone, as described in this article:

How to Configure Zones in Oracle Solaris 11 for Easy Cloning
Jeff McMeekin describes how to create a network topology of servers, routers, switches, and firewalls that you can clone right along with Oracle Solaris 11 zones.

However, if you are going to create several zones and perhaps configure them differently, why not make things easier on yourself? Why not prepare a few zone configuration plans? And when you're ready to create one, just push a button to execute one of the plans? This article by Laura Hartman describes how to do just that:

New!
How to Create Oracle Solaris 11 Zones with Oracle Enterprise Manager Ops Center 12c

Here's an overview of the process, lifted from the article:

"First, create an Oracle Solaris 11 zone profile and plan. The profile captures the zone configuration, including defining the storage and network details. The plan executes the configuration on selected targets. You can use and reuse the profile and plan to create zones with a consistent configuration.

"Then deploy the plan to create a new zone. When you deploy a plan, you identify the target operating systems and the number of zones to create. Before you submit the job to deploy the plan, you can modify some of the configuration details."

More info about Oracle Solaris 11 zones here:

- Rick

Website Newsletter Facebook Twitter

Wednesday Aug 15, 2012

It's Better with Btrfs

source

Two recently published articles to help you become proficient with the Btrfs file system in Oracle Linux:

How I Got Started with the Btrfs File System in Oracle Linux

By Margaret Bierman

Scalability and volume management. Write methodology and access. Tunables. Margaret describes these capabilities of the Btrfs file system, plus how it deals with redundant configurations, checksums, fault isolation and much more. She also walks you through the steps to create and set up a Btrfs file system so you can become familiar with it.

How I Use the Advanced Features of the Btrfs File System

By Margaret Bierman

How to create and mount a Btrfs file system. How to copy and delete files. How to create and manage a redundant file system configuration. How to check the integrity of the file system and its remaining capacity. How to take snapshots. How to clone. And more. In this article Margaret explores the more advanced features of the Btrfs file system.

Let us know what you think, and what you'd like to see Margaret write about in the future.

- Rick

Website Newsletter Facebook Twitter

Monday Aug 06, 2012

Basic and Advanced System Services Administration in Oracle Solaris 11

Does taming the behavior of your OS services manually make you feel less than your usual playful self? Lighten up. The Service Management Facility (SMF), introduced in Oracle Solaris 10 and extended in Oracle Solaris 11, provides the discipline those unruly services need. Here are two articles that will help get the most out of it.

Introducing the Basics of the Service Management Facility in Oracle Solaris 11

The SMF keeps track of the relationship between the services in your instance of Solaris. With this information, it can start services much more quickly at boot time, and it restart them automatically in the correct order if any of them fail. And that's only the beginning. In this article Glynn Foster explains what SMF does, and how to perform basic services administration with it, including how to use these four commands to get information about, and manage, your system services:

Command Description
svcadm Manage the state of service instances
svcs Provide information about services, including their status
svcprop Get information about service configuration properties
svccfg Import, export, and modify service configuration

Advanced Administration with the Service Management Facility in Oracle Solaris 11

In this article, Glynn Foster describes how to use some of the more advanced features of SMF, including service bundles, which you can use to deliver custom configuration across systems. And SMF profiles, which modify services to suit a particular installation. The introduction of layers in Oracle Solaris 11 provides better tracking of vendor-supplied customizations and administrative customizations for services and instances of services in four discrete layers, and site profiles, also described in this article, help you manage these layers more easily.

- Rick

Website Newsletter Facebook Twitter

Friday Aug 03, 2012

My Oracle RAC and Oracle Solaris Cluster Cheet Sheet

This gets complicated, so stop watching motoGP crash compilation videos for a sec.

We have Oracle Real Application Clusters (RAC). RAC lets you deploy a single Oracle Database across different servers. If the server in your Des Moines data center gets picked up by a tornado that hates you and dropped off in East Texas, the other servers pick up the load, and the database continues to operate without interruption. That's easy to understand.

We also have Oracle Solaris Cluster. It lets you deploy the Oracle Solaris operating system across different servers. If the server in your Barbados data center gets washed away by a hurricane that hates you and dropped off in West Africa, the other servers pick up the load, and the operating system continues to operate without interruption. A good quote:

White Paper: Extending Oracle Solaris for Business Continuity
"Oracle Solaris Cluster offers comprehensive and robust capabilities for keeping your business IT, including those running Oracle Database and Applications, up and running in the face of nearly every conceivable situation."

That's easy to understand, as well.

So why would somebody complicate our sysadmin lives by suggesting we install Oracle RAC on Oracle Solaris Cluster? What would that be, highly-available high availability?

Turns out that's not what they're suggesting. They're suggesting we install Oracle RAC not on Solaris Clusters, but on zone clusters. What's a zone cluster, you ask?

A zone cluster is a cluster created from Solaris zones that are physically located on different servers. That's similar to a regular cluster, but it uses zones instead of entire OS instances. Don't confuse a zone cluster with a failover cluster. Instead, read this white paper:

White Paper: Zone Clusters: How to Deploy Virtual Clusters and Why
This paper introduces the zone cluster, a virtual cluster in which an Oracle Solaris Zone is configured as a virtual node. The zone cluster supports the consolidation of multiple cluster applications on a single cluster.

That's all very interesting, but what about our original question:

Why would someone want to complicate our sysadmin lives by suggesting we install Oracle RAC on a zone cluster?

Turns out there two good reasons:

  • It's a better high-availability solution for a multi-tier application environment
  • It lets you isolate your database development, test, and deployment environments from each other.

How the Oracle RAC/Zone Cluster Combo Is Better For Multi-Tier Applications

Let's say that you are using your Oracle database as one tier in two different application environments. The first one is an HR application, the one second is an e-business suite. Both access the same database. Well, Oracle RAC would give you the high-availability for that database. But the applications would not be highly available. However, if you installed the database with Oracle RAC inside one zone cluster, and each application inside its own zone cluster, you'd make both application environments highly avaiable. And, if you limit the administrative privileges for each zone cluster, you'd get administrative isolation, as well.

How the Oracle RAC/Zone Cluster Combo Is Safer for Deployment

You've probably heard by now about Knight Capital Group's trading glitch that dropped the company's value by 50% in one day. I don't know exactly what happened, but I wonder if they didn't deploy either their development or their test environment instead of the one that was ready for prime time.

I suppose it's a sysadmin's duty to learn from another sysadmin's misfortune. So, if you divide your zone clusters into development, test, and deployment environments, you might have a better shot at avoiding a similar catastrophe. For example, install Oracle RAC with an Oracle DB into your development zone cluster, and keep it isolated from your test and deployment zone clusters. One sysadmin controls the development cluster. Another the test cluster. And the biggest, baddest sysadmin controls the deployment cluster. When the development environment is ready for testing, the test admin must OK the migration. That goes double for the deployment environment. And all the while, each environment remains highly available.

Resources

Turns out that Oracle and the portion of Oracle that was once Sun Microsystems have been collaborating on Oracle RAC/Solaris Cluster solutions for a long time. Customers like this approach so much that we just published three articles explaining how to do it. Each article covers a different version of the software:

Article RAC Version Solaris Version Cluster Version
How to Deploy Oracle RAC 11.2.0.2 on Oracle Solaris Zone Clusters 11.2.0.2 10 3.3
How to Deploy Oracle RAC 11.2.0.3 on Oracle Solaris Zone Clusters 11.2.0.3 10 3.3
How to Deploy Oracle RAC 11.2.0.3 on Oracle Solaris 11 Zone Clusters 11.2.0.3 11 4.0

And if you want more, we also have a page full of links to all our Solaris Cluster how-to articles and background white papers:

Where to find everything Solaris Cluster-related

Don't be the sysadmin who bankrupts your company in one day. Get educated.

- Rick

Website Newsletter Facebook Twitter

Friday Jul 27, 2012

What To Give Your Favorite Sysadmin on Sysadmin Day

Source

Happy Sysadmin Day.

As the site says ...

This is the day that all fellow System Administrators across the globe will be showered with expensive sports cars and large piles of cash in appreciation of their diligent work ..."

We understand that times are tight, so we don't expect you to buy your favorite sysadmin a Ferrari. That can wait till next year. But it wouldn't hurt to rent them one. Just for the weekend. To remind them what a weekend feels like.

- Rick

Website Newsletter Facebook Twitter

Wednesday Jul 25, 2012

OTN Sysadmin Day in Denver, Colorado

Source

Can you find the sysadmin in the picture?

You might be able to on Thursday, August 23rd. OTN is hosting its next Sysadmin Day in Denver, Colorado, and we can never keep track of our sysadmins. In a place as purty as Colorado, it will be even harder.

Note: The date on the link above is incorrect. The correct date is Thursday, August 23rd.

Our previous OTN Sysadmin Day in Santa Clara had almost 100 attendees. The one in Denver will have similar presentations, but we're adding some content on virtualization. Which we hope to expand into a third track in the future. As usual, Pavel Anni opens our OTN Sysadmin Days with a talk about Oracle's dual OS strategy. He explains why Oracle offers two operating systems, and summarizes the main features of each one. Then we split off into two different groups to get our hands on each OS.

One group gets their hands on the ZFS filesystem, virtualization capabilities, and security controls of Oracle Solaris.

The other group gets their hands on the package management tools, services, and runs levels of Oracle Linux, plus its volume management tools and the Btrfs filesystem.

Both groups learn by doing, using the hands-on labs similar to those on OTN's Hands-On Labs page. Why attend an event in person when you could simply work the labs on your own? Two reasons:

  1. Since you are away from the obligations of the data center, you get to focus on working the labs without interruption.
  2. You get help from Oracle experts and other sysadmins who are working on the same labs as you.
The event is free. Here's the agenda:

Time Session
8:00 am System Shakedown
9:00 am Oracle's Dual OS Strategy
 

Oracle Solaris Track

Oracle Linux Track

10:00 am HOL: Oracle Solaris ZFS HOL: Package Management and Configuration
11:30 am HOL: Virtualization HOL: Storage Management
1:00 pm Lunch / Surfing OTN
2:00 pm HOL: Oracle Solaris Security HOL: Btrfs filesystem
3:00 pm Presentation: Oracle Enterprise Manager Ops Center 11g
3:30 pm Presentation: Oracle VM Manager
4:00 pm Discussion: What are the most pressing issues for sysadmins today?
5:00 pm Get lost in the mountains.

- Rick

Website Newsletter Facebook Twitter

Monday Jul 23, 2012

My First Impression of Oracle University's On Demand Training

Source

I live in abject fear of lectures. I spent 12 years in and old fashioned Catholic School, complete with full uniform and wooden paddles. The first 8 years were a futile attempt to civilize me. During the remaining four years, the main thing I learned was how to sleep with my eyes open. And college wasn't much better. I don't know how I finished. I'm not even sure I finished. Instead, give me a few scraps of metal, a blow torch, and let me figure it out.

So when the folks from Oracle University offered to let me take one of their On-Demand online courses, I raised an eyebrow. Me? Are you sure? Maybe you should talk to Sister Mary Shackles, my high school principal.

But I decided to give it a try. After all, I am now a contributing member of society. I can probably pay attention for a few minutes without screaming. Holy Moly was I surprised. Hold still whilst I elucidate ...

Oracle University's Transition to Oracle Solaris 11 On-Demand Training - Course Format

Eric Siglin, the instructor, looked like he could head-butt me into the next building. If he'd been my high school principal I might have done better. Mister Siglin, which is how I'll refer to him so I remain in his good graces, has a background in Oracle Database, Solaris, Linux, and Oracle's Database Machine. Not bad.

Once you register for the course, you land in a dashboard of sorts that has three parts:

Selectable course outline

This one's pretty straightfoward ... a list of the course segments, and you can jump back and forth between them.

High-def video screen

Mister Siglin has a wicked black Fu-Manchu/white beard combo. And in full screen mode the resolution is good enough to verify that it's not a fake. When he needs to show you a screen, Mister Siglin simply replaces the video with a shot of the screen, and sometimes shows up live in the right corner of the screen.

As with those superbike crash compilations videos that I enjoy watching so much on YouTube, you can expand the window to full screen.

Scrolling Text Window

Below the screen is a scrolling text window that highlights the words as Mr Siglin speaks them. Reminds me of the Sing-Along-With-Mitch programs on American TV. You can turn off this feature with the little red lock icon a the top right of the text box, though I can't imagine why.

This is too cool: if you want to go back and review a portion of the lecture, you can click on the text below the window, and the video rewinds to the part where the instructor, Mister Siglin, spoke that word. And it advances normally from there.

But wait! There's more. Enter a word into the search window, and the progress bar indicates where in the recording Mister Siglin has said that word. Click on the indicator, and the video rewinds to that spot. Along with the scrolling text, of course. Unless you're the kind of guy who turns off the cool scrolling text. You probably pay for your fast food with small coins, don't you?

Course Content

As cool as all those bells and whistles are, the best part is the content. Here's an example of Mister Siglin's introductory comments.

"We are assuming that you have some prior Solaris experience coming in here, because we're going to address what's new with Solaris. We're going to talk about the image packaging system. Now, the image packaging system reminds me an awful lot of what we have in the Linux environment. The automated installer, which is a replacement for Jumpstart...

"Plus, we're also going to come up with some ideas to help it make it easier for you to transition from Solaris 10 to Solaris 11...

"So we're going to look at managing the software packages in Solaris 11. And that's going to continue perhaps until tomorrow. That's one of the nice things about having a small group like this one, that makes our schedule a little more flexible. So then we're going to talk about enhancements to the installation process. We have a couple of different ways of looking at that, because the installer's been improved. We have several options. And then we're going to get into Solaris Zones. We're going to take a look at what is new with the Solaris Zones, new with networking, especially since we're dealing with a lot more virtualization. And then on the last day, we're going to get into storage enhancements. There are some major enhancements with ZFS, for example. We're going to address those. And then the security enhancements that are in this version of Solaris.

If you get a minute ...

In a couple of weeks I'll tell you what I think about what I've been learning. Till then, here's another motorcycle crash video. And, for those of you who have not surrendered the romance in your soul to the rigors of keeping an IT shop humming, here's another enthusiastic sing-a-long from Mitch.

- Rick

Website Newsletter Facebook Twitter

Tuesday Jul 17, 2012

How to Protect Your Oracle Linux System from the Higgs Boson

Now that the Higgs Boson particle has been gently coaxed out of hiding, you know what's gonna happen, don't you? Your boss is gonna walk into your office and demand a plan for protecting your Oracle Linux system against it.

You could act like a smart aleck sysadmin and inform him or her that it took a team of scientists 10 years and 500 trillion collisions to get conclusive evidence of its existence, and let's not even talk about how difficult it was for God to create the elusive thing, but that would violate the first law of corporate survival:

Never, ever make your boss look stupid

Instead, jump out of your chair and say "OMG! I hadn't though of that!" Then read our latest article and use what you learn to write up a plan that will make your boss look real good to his or her boss. (Just make sure your name appears nowhere.)

Tips for Hardening an Oracle Linux Server

Lenz Grimmer and James Morris provide guidelines for:

  • Minimizing the software footprint
  • Minimizing active services
  • Locking down network services
  • Disabling or tightening use of SSH
  • Configuring mounts, file permissions, and ownerships
  • Managing Users and Authentication
  • Other Security Features and Tools
  • Cryptography
I hope you enjoy reading the article as much as I did. And good luck with your career.

- Rick

Website Newsletter Facebook Twitter

Thursday Jun 28, 2012

Similar But Not The Same

A few weeks ago we published an article that explained how to use Oracle Solaris Cluster 3.3 5/11 to provide a virtual, multitiered architecture for Oracle Real Application Cluster (Oracle RAC) 11.2.0.2. We called it ...

How to Deploy Oracle RAC on Zone Clusters

Welllllll ... we just published another article just like it. Except that it's different. The earlier article was for Oracle RAC 11.2.0.2. This one is for Oracle RAC 11.2.0.3. This one describes how to do the same thing as the earlier one --create an Oracle Solaris Zone cluster, install and configure Oracle Grid Infrastructure and Oracle RAC in the zone cluster, and create an Oracle Solaris Cluster resource for Oracle RAC-- but for version 11.2.0.3 of Oracle RAC. Even though the objective is the same, and the version is only a dot-dot-dot release away, the process is quite different. So we decided to call it:

How to Deploy Oracle RAC 11.2.0.3 on Zone Clusters

Hope you can keep the different versions clear in your head. If not, let me know, and I'll try to make them easier to distinguish.

- Rick

Website Newsletter Facebook Twitter

Tuesday Jun 12, 2012

Whole Lotta Virtualization Goin' On

Lately we've published a lot of content about virtualization. Here's a sampling.

Podcat: Technology Preview of Transcendent Memory

Turns out that in a virtual environment, RAM is the bottleneck. Not because it's slow, it's not, but because each CPU still had to use its own RAM. Which gets expensive. In this podcast, Dan Magenheimer describes how Oracle and the open source community taught the guest kernel in Oracle Linux to share its memory with other CPU's. Transcendent memory will wind up saving large data centers a lot of money. Find out how.

Tech Article: How to Use Oracle VM Templates

This article describes how to prepare an Oracle VM environment to use Oracle VM Templates, how to obtain a template, and how to deploy the template to your Oracle VM environment. It also describes how to create a virtual machine based on that template and how you can clone the template and change the clone's configuration.

Tech Article: How to Set Up a Load Balanced Application Across Two Oracle Solaris Zones

Install Apache Tomcat on two Oracle Solaris zones. Connect them across a VPN. And let the Integrated Load Balancer in Oracle Solaris 11 manage traffic. Presto: high(er) availability in a single server.

Tech Article: How to Install Oracle RAC on Oracle Solaris Zone Clusters

Learn how to implement a multi-tiered database environment that isolates database tiers and administrative domains, while taking advantage of centralized (and simpler) cluster admin.

For fans of Jerry Lee Lewis

If you're a fan of Jerry Lee Lewis, you might enjoy this video.

- Rick

Website Newsletter Facebook Twitter

Thursday May 24, 2012

Overcoming Your Fear of Repositories

One of the best features of Oracle Solaris 11 is its software update model. As you have probably heard many times by now, the Image Packaging System (IPS) handles package dependencies automatically, so you no longer have to check them manually or create scripts that assemble the correct set of packages .

If you don't have a support contract, you have to wait until the next release of Oracle Solaris 11 to get the latest updates. But if you do have a support contract, you can keep your system updated with the latest security updates and bug fixes by downloading updates from the Oracle Support Repository. We recently published two articles that describe how, plus one more that shows you how to create multiple internal repositories.

How to Update Oracle Solaris 11 Systems from the Oracle Support Repository, by Glynn Foster

You may already know that you don't have to worry about manually tracking and validating patch dependencies when you update a version of Oracle Solaris 11. But did you also realize you can apply the updates to an alternate boot environment, and then schedule the switchover to happen automatically at a later time? Glynn Foster explains how, and how to make sure everything went well.

More Tips for Updating Oracle Solaris 11 Systems from the Oracle Support Repository, by Peter Dennis

The Oracle Support Repository contains bug fixes and critical security fixes that can be applied to existing Oracle Solaris 11 installations between major releases. The repository is updated monthly. Peter Dennis describes how to access those updates and apply them to your systems.

How to Create Multiple Internal Repositories for Oracle Solaris 11, by Albert White

Even though you may get all your software updates to Oracle Solaris 11 from an external repository, you may still want to create different internal repositories to serve different versions of Oracle Solaris 11 to different types of systems. Albert White shows you how to create and manage internal repositories for release, development, and support versions of Solaris 11.

There's plenty more where these came from. Be sure to bookmark our Installation Spotlight page, maintained by the kind and prolific folks who bring you Oracle Solaris 11.

About the picture ...
Laird Hamilton is a god.
Teahupoo is a killer wave.
Laird owned it.
Be like Laird.

- Rick

Website Newsletter Facebook Twitter
About

Contributors:
Rick Ramsey
Kemer Thomson
and members of the OTN community

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Blogs We Like