The Dangers of “Recipes”
By Robert Freeman-Oracle on Aug 01, 2014
The Story of the Would Be Baker
There was once a man who loved to bake. He lived on the shore line. He would love to watch the ocean from his windows as he would bake his wares, steadfastly following his recipes. The recipes were ones that had been handed down from his mother and his mother’s mother, honed and crafted over generations of trial and error. The result of these recipes was the creation of amazing masterpieces of baking. Pies, cakes, rolls and other breads that he cooked were simply amazing. His mother taught him how to bake, and how to follow the recipes and he had eventually opened his bakery. He was known for miles around as an amazing baker, and people came from miles around to buy and eat his baked goods.
One day, there was a woman had heard of this man and his wonderful baking. She drove to the bakery to sample his wares. She ordered several items and as she ate them she fell in love with his baking. In her life, she had cooked a great deal and always wanted to try baking. Her few attempts had not be the greatest, but she always felt that was because the recipes were not the best. Here, there was no mistaking that these recipes were the best recipes and they were tested and tried over time. While she had little experience with baking, she felt confident that with these recipes in hand she would soon be an amazing baker too. She had wonderful memories as a child of her mother taking her to the bakery and as a result, opening a bakery was a dream of hers. She knew with these recipes that she could not fail.
Excited, she went to the man, and offered to buy his recipes so she could open her own bakery. Staring at a check with several zero’s in the number, the man agreed and handed a copy of his time honored family baking recipes to the woman. The woman was elated and got in her car and drove home, some three hundred miles away and in a beautiful little city nestled high in the mountains. The city was a famous ski resort and she was excited to open her bakery and sell wonderful wares to the tourists who would soon be entering her town . Quickly she located a shop front that already had all the baking equipment in it that she needed.
Several days later, having acquired the best of ingredients, and having followed the mixing instructions to the letter, she started to bake the recipes. She started baking cookies and pies and all manners of bread. The smell filled the room, and she found herself lost in the delightful aromas as they proceeded to bake. In those delightful smells, she was content that all was well. She peeked in the ovens from time to time, and all seemed to be going well.
The first timer went off. It was the timer for the cookies. She remembered these cookies from the bakery – Oatmeal Raisin cookies. The woman, excited to experience the work of her own hands, opened the over where the trays of cookies were and pulled each tray out excitedly – with a smile on her face. Yet, when she looked at the cookies as she pulled them out, there was something that didn’t seem quite right about them. She could not put her finger on what it was, and she decided it was just that they needed to cool down. She put the trays down and waited as the cookies cooled. Finally, she looked at the cookies, which had cooled down – still, something didn’t seem right.
She picked up the cookie and put it in her mouth. When she bit down on the cookie, it was dry and crumbly and nothing like the chewy and delicious cookie she had eaten earlier. As each timer went off, and she removed the item she was baking, she kept finding that something was off in each one. Some of the breads didn’t set correctly, cakes were not moist – nothing came out right.
What was the problem?
The Problem with Recipes
Did you notice in the story that the man’s bakery was at sea level and that the woman’s new bakery was up in the mountains, at altitude? This was the problem with the recipes. Because the environment had changed, the recipe was no longer able to produce the wonderful baked goods that the man had made. My wife tells me that baking is a science, and that cooking is an art. Baking requires the right mixture of all of the elements – flour, water, sugar, heat, time and so on. A recipe is a method of putting all these elements together given some very specific base variables, altitude being one of those. If you change any of these variables, then you need to understand the science behind the recipe to adjust the recipe. For example, when baking at higher altitude you will often cook at a higher temperature for a shorter period of time. This can lead to evaporative issues in the item being cooked, so you will often need to add some additional water to keep the resulting baked goods from being too dry.
There are many variables, and dependencies (such as the higher heat requiring more water to avoid dryness) that will cause a failure of the recipe if you are not aware of them and adjust accordingly. Miss one adjustment and the whole affair becomes a sad memory to what could have been. In cooking, we have the opportunity to try and try again, adjusting the recipe until it’s just right again. With production databases – we don’t usually get to many redo’s when it comes to recovering from data loss. Loose data and you better get that word processor ready and update your resume.
Why Recipes are Dangerous in IT
There are a number of books published that offer “recipes” to do this and that in an Oracle database. Often in user groups I find people asking lots of “How do I” kinds of questions (which is understandable) and I see much fewer “How does this work” kind of questions. To me, this demonstrates a fundamental flaw in the thinking of IT professionals these days. In days past, you had to know how something worked in order to use it properly. Often the only source of help that you had was poorly written documentation, so we were often forced to learn from the ground up – learning how something worked and then applying that knowledge to making that something work.
In those days, we would work from the statement: “I need to do X.” – for example, “I need to be able to recovery my data.”. Based on the statement (which might also be called a requirement), we designed a solution understanding the technology that we were working with. That is to say, we didn’t often have tools that would do the work for us, so we had to build those tools. In having to build those tools, we had to understand the underlying principles of how something worked.
This was not a very scalable solution of course, everyone writing their own tools. Now, we have wonderful tools like RMAN that make our lives much easier, and provide for us the ability to focus on other work. This is a wonderful thing.
However, I have noticed what I think is an unintended consequence of the advent of these new tools. This consequence is that we no longer are forced to understand the technology that the tool is abstracting us from. In an effort to quickly get up to speed and implement the tool, we run to recipes, or tutorials, to help us quickly setup and use the tool. Indeed, we clamor for these recipes because of the growing complexity of these tools. We want to shortcut the learning curve because it is so very steep. We then dismiss the notion of learning about the internal functioning of the tool, and depend on someone else’s knowledge and experience to lead the way, following their recipes.
However, just as the man in our story didn’t realize that altitude would impact his recipes (for he never really professionally learned baking, only to read and follow his family recipes) we often don’t understand how to adjust the recipes when our environment changes. Perhaps the recipe for backing up does not work just right, and we struggle to figure out how to make it work. Worse, maybe the recipe for recovery does not work right, and we only learn about this problem when it’s crunch time and we have to restore and recovery our database.
The point is that just asking for and following a recipe is not enough for a professional DBA. You are expected to understand how things work, not just how to make things work in a specific set of pre-defined situations. Just understanding recipes is not enough when it comes time to architect your solution because the permutations of possible architectures and methodologies is huge. A number of things from licensing, software versions, clustering, DR solutions, business rules and many more variables can take a recipe and make it unusable. The problem is if you don’t understand the technology, you really don’t know what you’re getting when you just follow a recipe. You don’t know that you’re not meeting a business need until it’s too late.
So, What to do Instead?
The solution to the problem, I think, is as follows:
· Deeper Education – You need to take the time and understand how things work. In the case of backup and recovery – you need to know how the database works, and what it does as transactions are processed. You need to know how these mechanisms are used during the recovery process. Do you find that you have to return to the recipes time and time again because you don’t know how to do things off the top of your head? Are you using all of the RMAN features to your best advantage? If your just following recipes, you can’t possibly know these things.
- Understand Architecture – Sure, maybe you know RMAN, but what don’t you know? Do you know how to implement a tiered backup infrastructure with maintained retention? Do you know how to test restores both with and without actually restoring the database? Is the recipe you are using really the one best suited to your needs? Do you understand all of the different backup and restore methodologies and which one will best meet your needs?
- Define SLA’s – The term SLA must be one of the dirtiest words in IT because every time I use it people’s heads bow, I hear gasps of air and fingers start pointing everywhere. How can you define anything without an understanding and agreement between all of the stakeholders as to what you are going to do. Recipes are not SLA’s.
- Define and understand your strategy – Knowing where you need to go is important. A recipe cannot understand what the business needs and requirements are – therefore a recipe can not be a blueprint for your organizations strategy. A recipe is only a small part of the whole of what needs to happen.
- Define best practices and standards – Neither of these two terms are synonymous with recipes. Instead, they define the best way of doing things and they provide a common, standardized and scalable way of doing things. Best practices and standards are borne from education, understanding and application of SLA’s. They may result in more than one standardized way of doing things – for example your standard backup and recovery strategy for OLTP databases might be different than that for data warehouse. The thing that best practices and standards to is to put together a scalable and repeatable and tested solution for your enterprise while also acknowledging that it needs to be flexible.
- Mentoring, monitoring, control and communication – these are critically important items, and I’ve rarely if ever seen these called out in a recipe.
My belief that understanding how something works is paramount to understanding how to implement solutions that meet specific requirements is why I spend the first several chapters of my RMAN book explaining how Oracle works with respect to backup and recovery. Some get perturbed by that, suggesting that this method takes too long to get started with RMAN. I acknowledge that understanding the basics first does take time (and in fact in my new addition I have added a quick start guide) but at the end of the day the difference between rank amateur and professional is the degree of understanding of that which you profess expertise about. (and if you ever hope to pass a technical interview with me, you best be able to describe in general terms what happens to the database during a restore and recovery, down to how UNDO and redo are applied, and what happens to uncommitted transactions and even the question – will blocks that contain uncommitted data ever be flushed to the database datafiles by DBWR and if so, why?)
At the end of the day, the needs and requirements of an enterprise are way beyond the scope of simple recipes. Recipes are not best practices and often they are not even complete examples of how to do something. They are simply a means of trying to educate, and in my mind it’s a poor approach to education. Why? Because far too many people believe that recipes are the way that something should be done, instead of understanding that, at best, it’s just one way of doing what should be done.
Way too often, I’ve had people point at a recipe, or an example and claim that’s the way it has to be done. Way too often I’ve looked at backup scripts and asked people “Where did you source this piece of junk from?” (I’m usually nicer than that) only to find that it’s been cut and pasted from this book , that book or some website. Way too often, as we dive into backup and recovery discussions do I find that their methodology does not meet their requirements and, once in a while, the scariest moments are when I realize that their backups are wholly unrecoverable – and that the DBA’s never realized that.
Way to often I’ve asked questions of DBA’s about restore situations that throw them way off their restore “scripts” or recipes. Way too often, when things go off script (as happens with restores very often) the DBA gets lost. I don’t expect you to understand how to force a database open, indeed that information is not documented officially. I do expect you to tell me that when a datafile or two is lost that restoring the datafiles online is a far better situation than shutting down the database and restoring the whole thing. (true story by the way).
When the time comes for an actual restore and recovery, those are scary moments indeed. The fight and flight response will be in full swing and you need to be prepared – and you need to understand – how the database works, and how the restore and recovery process works so you can get the database back up and running with a minimum of outage.
In the end if we are managing databases, and are charged to protect the data within them, then recipes are not the way to ensure that this occurs. Only good old education and understanding will suffice. That takes time, commitment, prioritization, determination and a refusal to be lazy. Otherwise, you might as well keep that resume up to date – for your time is coming.
And you don’t want to sit in a job interview, trying to answer the question, “Why did you leave your last job?” when the correct answer is, “Because I cratered the database by issuing an rm –r * from root, and in trying to restore the database realized that I had killed all of the online redo logs and that the database could only be restored and recovered to a point in time 2 hours before I trashed the file systems.”
That is a bad day my friends. How do you feel about recipes?
Note: Edited in a couple of places for grammar and content since original post.