• August 1, 2014

The Dangers of “Recipes”

Guest Author

The Story of the
Would Be Baker

There was once a man who loved to bake. He lived on the
shore line. He would love to watch the ocean from his windows as he would bake
his wares, steadfastly following his recipes. The recipes were ones that had
been handed down from his mother and his mother’s mother, honed and crafted
over generations of trial and error. The result of these recipes was the
creation of amazing masterpieces of baking. Pies, cakes, rolls and other breads
that he cooked were simply amazing. His mother taught him how to bake, and how
to follow the recipes and he had eventually opened his bakery. He was known for
miles around as an amazing baker, and people came from miles around to buy and eat
his baked goods.

One day, there was a woman had heard of this man and his wonderful baking. She drove to
the bakery to sample his wares. She ordered several items and as she ate them she fell in love with
his baking. In her life, she had cooked a great deal and always wanted to try
baking. Her few attempts had not be the greatest, but she always felt that was
because the recipes were not the best. Here, there was no mistaking that these
recipes were the best recipes and they were tested and tried over time. While
she had little experience with baking, she felt confident that with these
recipes in hand she would soon be an amazing baker too. She had wonderful
memories as a child of her mother taking her to the bakery and as a result,
opening a bakery was a dream of hers. She knew with these recipes that she
could not fail.

Excited, she went to the man, and offered to buy his recipes
so she could open her own bakery. Staring at a check with several zero’s in the
number, the man agreed and handed a copy of his time honored family baking recipes
to the woman. The woman was elated and got in her car and drove home, some
three hundred miles away and in a beautiful little city nestled high in the
mountains. The city was a famous ski resort and she was excited to open her
bakery and sell wonderful wares to the tourists who would soon be entering her
town . Quickly she located a shop front that already had all the baking
equipment in it that she needed.

Several days later, having acquired the best of ingredients,
and having followed the mixing instructions to the letter, she started to bake
the recipes. She started baking cookies and pies and all manners of bread. The
smell filled the room, and she found herself lost in the delightful aromas as
they proceeded to bake. In those delightful smells, she was content that all
was well. She peeked in the ovens from time to time, and all seemed to be going

The first timer went off. It was the timer for the cookies.
She remembered these cookies from the bakery – Oatmeal Raisin cookies. The
woman, excited to experience the work of her own hands, opened the over where
the trays of cookies were and pulled each tray out excitedly – with a smile on
her face. Yet, when she looked at the cookies as she pulled them out, there was
something that didn’t seem quite right about them. She could not put her finger
on what it was, and she decided it was just that they needed to cool down. She
put the trays down and waited as the cookies cooled. Finally, she looked at the
cookies, which had cooled down – still, something didn’t seem right.

She picked up the cookie and put it in her mouth. When she
bit down on the cookie, it was dry and crumbly and nothing like the chewy and
delicious cookie she had eaten earlier. As each timer went off, and she removed
the item she was baking, she kept finding that something was off in each one.
Some of the breads didn’t set correctly, cakes were not moist – nothing came
out right.

What was the problem?

The Problem with

Did you notice in the story that the man’s bakery was at sea
level and that the woman’s new bakery was up in the mountains, at altitude?
This was the problem with the recipes. Because the environment had changed, the
recipe was no longer able to produce the wonderful baked goods that the man had
made. My wife tells me that baking is a science, and that cooking is an art.
Baking requires the right mixture of all of the elements – flour, water, sugar,
heat, time and so on. A recipe is a method of putting all these elements
together given some very specific base variables, altitude being one of those.
If you change any of these variables, then you need to understand the science
behind the recipe to adjust the recipe. For example, when baking at higher
altitude you will often cook at a higher temperature for a shorter period of
time. This can lead to evaporative issues in the item being cooked, so you will
often need to add some additional water to keep the resulting baked goods from
being too dry.

There are many variables, and dependencies (such as the
higher heat requiring more water to avoid dryness) that will cause a failure of
the recipe if you are not aware of them and adjust accordingly. Miss one
adjustment and the whole affair becomes a sad memory to what could have been.
In cooking, we have the opportunity to try and try again, adjusting the recipe
until it’s just right again. With production databases – we don’t usually get
to many redo’s when it comes to recovering from data loss. Loose data and you
better get that word processor ready and update your resume.

Why Recipes are
Dangerous in IT

There are a number of books published that offer “recipes”
to do this and that in an Oracle database. Often in user groups I find people
asking lots of “How do I” kinds of questions (which is understandable) and I
see much fewer “How does this work” kind of questions. To me, this demonstrates
a fundamental flaw in the thinking of IT professionals these days. In days
past, you had to know how something worked in order to use it properly. Often
the only source of help that you had was poorly written documentation, so we
were often forced to learn from the ground up – learning how something worked
and then applying that knowledge to making that something work.

In those days, we would work from the statement: “I need to
do X.” – for example, “I need to be able to recovery my data.”. Based on the
statement (which might also be called a requirement), we designed a solution
understanding the technology that we were working with. That is to say, we didn’t
often have tools that would do the work for us, so we had to build those tools.
In having to build those tools, we had to understand the underlying principles
of how something worked.

This was not a very scalable solution of course, everyone
writing their own tools. Now, we have wonderful tools like RMAN that make our
lives much easier, and provide for us the ability to focus on other work. This
is a wonderful thing.

However, I have noticed what I think is an unintended consequence
of the advent of these new tools. This consequence is that we no longer are
forced to understand the technology that the tool is abstracting us from. In an
effort to quickly get up to speed and implement the tool, we run to recipes, or
tutorials, to help us quickly setup and use the tool. Indeed, we clamor for
these recipes because of the growing complexity of these tools. We want to
shortcut the learning curve because it is so very steep. We then dismiss the
notion of learning about the internal functioning of the tool, and depend on
someone else’s knowledge and experience to lead the way, following their

However, just as the man in our story didn’t realize that
altitude would impact his recipes (for he never really professionally learned
baking, only to read and follow his family recipes) we often don’t understand
how to adjust the recipes when our environment changes. Perhaps the recipe for
backing up does not work just right, and we struggle to figure out how to make
it work. Worse, maybe the recipe for recovery does not work right, and we only
learn about this problem when it’s crunch time and we have to restore and
recovery our database.

The point is that just asking for and following a recipe is
not enough for a professional DBA. You are expected to understand how things
work, not just how to make things work in a specific set of pre-defined
situations. Just understanding recipes is not enough when it comes time to
architect your solution because the permutations of possible architectures and
methodologies is huge. A number of things from licensing, software versions,
clustering, DR solutions, business rules and many more variables can take a
recipe and make it unusable. The problem is if you don’t understand the technology,
you really don’t know what you’re getting when you just follow a recipe. You
don’t know that you’re not meeting a business need until it’s too late.

So, What to do

The solution to the problem, I think, is as follows:

· Deeper Education – You need to take the time and
understand how things work. In the case of backup and recovery – you need
to know how the database works, and what
it does as transactions are processed. You need to know how these mechanisms
are used during the recovery process. Do you find that you have to return to
the recipes time and time again because you don’t know how to do things off the
top of your head? Are you using all of the RMAN features to your best
advantage? If your just following recipes, you can’t possibly know these

  • Understand Architecture – Sure, maybe you know
    RMAN, but what don’t you know? Do you know how to implement a tiered backup
    infrastructure with maintained retention? Do you know how to test restores both
    with and without actually restoring the database? Is the recipe you are using
    really the one best suited to your needs? Do you understand all of the
    different backup and restore methodologies and which one will best meet your
  • Define SLA’s – The term SLA must be one of the
    dirtiest words in IT because every time I use it people’s heads bow, I hear
    gasps of air and fingers start pointing everywhere. How can you define anything
    without an understanding and agreement between all of the stakeholders as to
    what you are going to do. Recipes are not SLA’s.
  • Define and understand your strategy – Knowing where
    you need to go is important. A recipe cannot understand what the business needs
    and requirements are – therefore a recipe can not be a blueprint for your organizations
    strategy. A recipe is only a small part of the whole of what needs to happen.
  • Define best practices and standards – Neither of
    these two terms are synonymous with recipes. Instead, they define the best way
    of doing things and they provide a common, standardized and scalable way of
    doing things. Best practices and standards are borne from education,
    understanding and application of SLA’s. They may result in more than one
    standardized way of doing things – for example your standard backup and
    recovery strategy for OLTP databases might be different than that for data
    warehouse. The thing that best practices and standards to is to put together a
    scalable and repeatable and tested solution for your enterprise while also
    acknowledging that it needs to be flexible.
  • Mentoring, monitoring, control and communication
    – these are critically important items, and I’ve rarely if ever seen these
    called out in a recipe.

My belief that understanding how something works is
paramount to understanding how to implement solutions that meet specific
requirements is why I spend the first several chapters of my RMAN book
explaining how Oracle works with respect to backup and recovery. Some get
perturbed by that, suggesting that this method takes too long to get started
with RMAN. I acknowledge that understanding the basics first does take time
(and in fact in my new addition I have added a quick start guide) but at the end
of the day the difference between rank amateur and professional is the degree
of understanding of that which you profess expertise about. (and if you ever
hope to pass a technical interview with me, you best be able to describe in
general terms what happens to the database during a restore and recovery, down
to how UNDO and redo are applied, and what happens to uncommitted transactions
and even the question – will blocks that contain uncommitted data ever be
flushed to the database datafiles by DBWR and if so, why?)

At the end of the day, the needs and requirements of an
enterprise are way beyond the scope of simple recipes. Recipes are not best practices
and often they are not even complete examples of how to do something. They are
simply a means of trying to educate, and in my mind it’s a poor approach to education.
Why? Because far too many people believe that recipes are the way that
something should be done, instead of understanding that, at best, it’s just one
way of doing what should be done.

Way too often, I’ve had people point at a recipe, or an
example and claim that’s the way it has to be done. Way too often I’ve looked
at backup scripts and asked people “Where did you source this piece of junk
from?” (I’m usually nicer than that) only to find that it’s been cut and pasted
from this book , that book or some website. Way too often, as we dive into
backup and recovery discussions do I find that their methodology does not meet
their requirements and, once in a while, the scariest moments are when I
realize that their backups are wholly unrecoverable – and that the DBA’s never
realized that.

Way to often I’ve asked questions of DBA’s about restore
situations that throw them way off their restore “scripts” or recipes. Way too
often, when things go off script (as happens with restores very often) the DBA
gets lost. I don’t expect you to understand how to force a database open,
indeed that information is not documented officially. I do expect you to tell
me that when a datafile or two is lost that restoring the datafiles online is a
far better situation than shutting down the database and restoring the whole
thing. (true story by the way).

When the time comes for an actual restore and recovery,
those are scary moments indeed. The fight and flight response will be in full
swing and you need to be prepared – and you need to understand – how the
database works, and how the restore and recovery process works so you can get
the database back up and running with a minimum of outage.

In the end if we are managing databases, and are charged to
protect the data within them, then recipes are not the way to ensure that this
occurs. Only good old education and understanding will suffice. That takes time,
commitment, prioritization, determination and a refusal to be lazy. Otherwise,
you might as well keep that resume up to date – for your time is coming.

And you don’t want to sit in a job interview, trying to
answer the question, “Why did you leave your last job?” when the correct answer
is, “Because I cratered the database by issuing an rm –r * from root, and in
trying to restore the database realized that I had killed all of the online
redo logs and that the database could only be restored and recovered to a point
in time 2 hours before I trashed the file systems.”

That is a bad day my friends. How do you feel about recipes?

 Note: Edited in a couple of places for grammar and content since original post.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.