By MT:15 on Aug 14, 2007
Summertime tends to be the time of year when people naturally slow down. (In some cases, it is because of absolutely unbearable heat -- who wants to move fast when it's 98 degrees out?) Summer is the season when you take vacations, change gears and get away from work. You realize there is a big wide world out there that comes to you through other vectors than email and the Internet. Even if you are working, there are a lot of distractions, like warm summer nights, sun-dappled days for bike rides, warm water and late sunsets for surfing. Summer is just one big speed bump telling you to slow down, observe, to pay attention to something other than the daily commute. You need those speed bumps.
I was out surfing recently at my usual surf habitat in Pacifica. Normally, you catch a wave, the face slopes; you slot into it and keep riding the face of the wave. My best surfing buddy, Kerry, calls me "the Queen of Trim," because when I catch a wave, I am really good at finding the absolutely right spot on the board to keep it in perfect trim: slotted into the wave so I can keep riding until there is no more wave. In my opinion, "kicking out" before the ride is done is the 8th deadly sin and a waste of a perfectly good wave.
The local surf break I frequent has a bit at the north end of the beach that's kind of steep, so depending on the tides, you may be riding in and find that you meet the backwash of the previous wave -- or two. Those backwash bumps are really a surprise when you hit them, because you end up riding UP the back of a wave and down the front again, sometimes more than once. Fun. Strange. Shakes up the surf session. Surfing is not, in general, supposed to be like riding a roller coaster. Except when it is. You remember not to surf on autopilot or you will wipe out when you hit the "speed bump." It's the ocean's way of getting you to pay attention.
Some of the speed bumps I get are in my inbox: I hear from people I haven't heard from in awhile, I end up being distracted from my "regular work," only the distraction turns out to be pretty important. I got an email recently from a colleague and friend -- Jaime Chanaga. I was impressed with Jaime the first time I met him (I think we were on a panel together at some security-fest or another). Jaime is among the few -- the very few -- who, as a CISO several years ago was already putting questions in his RFPs asking about the kind of care his vendors took in the way they built security into products.
Anyway, Jaime recently emailed me with a PPT he had done on security excellence, which, in my own nosy parker way, I suggested he turn into a blog (he did). His security excellence principles include things like valuing people and leading with integrity. Why is his blog a good speed bump? Because you could read an endless procession of Management Tomes without ever finding useful advice like this. His advice is good, solid, and timeless. I know this because I spent two years getting an expensive graduate business degree from Wharton and I am not really sure that most people there know or care about the difference between management and leadership. (Unfortunately, too many people who espouse "principle-centered leadership" -- or whatever the latest business buzzword is -- are not practitioners of it: "look after people" really means "look after (self; by using) people."
Since I know him, I can say with confidence that Jaime practices what he preaches. His suggestions for security excellence are good reminders for those days when you are feeling like crankily cutting corners with people you work with or who work for you. Thank you, Jaime, for a speed bump reminder -- your email -- that being a decent and good person of high integrity is among the most valuable business skills there are, for security gurus and others. (These principles are mom-approved, too.)
I am playing fast and loose with the term "speed bump," but I would like to extend it to various other cautionary signs that admonish some attention on the road from where you've been to where you are going. I add that some of these signs have real meaning here in Idaho.
"Game crossing," for example, does not mean a bunch of geeks with X-boxes are likely to cross Highway 20, no sirree. Not only do "Game Crossing" signs in Idaho mean that herds of critters -- like elk -- move along various corridors that cross highways, the state of Idaho adds a flag to the Game Crossing signs during the seasons when the animals are most active. It's Idaho Fish and Game's way of saying, "Pay attention: we really mean it!" I just read a blog entry by the former police chief of Hailey, Idaho talking about near misses with large critters. His theory is that suicidal and vengeful deer cross the highway to target people who have hunting licenses. (I feel compelled to add that Brian's blog entry provides a far better sense of a road trip across America than most anything else I have read.)
Game crossing signs, aside from mitigating the risk of "bull elk as unwanted hood ornament," do help you slow down and look at the scenery, which in Idaho is pretty spectacular. On the road from Boise to Ketchum, I've seen elk, moose, deer, antelope, peregrine falcons (Idaho is the home of the World Center for Birds of Prey, and has done more than any other state to help bring the peregrine falcon back from near-extinction), coyote, skunk, owls, porcupine, sand hill cranes, and foxes. There are wolves in the northern part of the Wood River Valley, though I've never seen them. Only last night, I saw a mother mule deer and twin fawns crossing Sun Valley Road at dusk. I hope I never get jaded at seeing such amazing animals. (It sure beats doing that 512th email of the day.)
The other speed bumps that really mean something around here are the fire warning signs. This summer has been unusually hot, dry, and windy. A perfect (fire) storm. Large parts of the west are burning, in some cases because of lightning strikes, in other cases because of stupid and careless individuals. A sign in a campground that says "No Campfires" means no campfires. "Please do not set off illegal fireworks," means it, too. The first fire of the summer here in Blaine County burned one of my favorite walks in Sun Valley -- Trail Creek. The entire mountain is blackened and looks like a moonscape. All because someone carelessly set a fire during high burn conditions. If you are camping or hiking in the west this summer, be careful, folks. Fire warning signs really mean something this year and there is no Undo button if you get it wrong.
Since I happily set up a topic on cautionary notes/speed bumps, I am going to add one myself, and it goes to the always contentious, World-Wrestling-Federation-has-never-seen-anything-like-it area of how to handle security vulnerabilities. I'm going to avoid the pothole -- for now -- of responsible disclosure vs. full disclosure except to note that, like everything else in life, there is no perfect disclosure policy that optimizes on all parameters. And in fact, there are no perfect patching policies, either. Only in Never- Never Land or Oz can you patch every single security vulnerability of every single severity in real time on all affected versions such that patch application is real time and perfect, too. That should be obvious -- when we get into these discussions -- but it isn't.
The gritty reality is that life is constrained. No company or organization or individual has infinite resources ("resource" includes time, expertise, people, pretty much anything that you need to effect positive change but doesn't grow on trees). By definition, something that is not infinite or free -- or both -- is constrained. For example, I don't have to pay to use the ocean, but my surfing is constrained by the number of other people out in the water, the time between swells (sometimes it's a good 20 minutes between waves, some days it's more like "catch a wave, paddle out, catch another wave right away, come in after 45 minutes because you are exhausted") and so on. Even surfing is constrained because while there are an infinite number of waves -- over time -- there are only so many during the time I am out surfing and each wave holds two people at most.
We shouldn't always attribute evil motives to the fact that organizations live with constraints. Constraints affect both vendors and their customers. Vendors cannot always create patches for every single issue on every single old version of product (sometimes, a fix would require an architectural change, which we all know can't happen on old products in all cases since architectural fixes aren't always backportable). Constrained resources also apply to the companies who apply patches. At a minimum, companies need their systems to be up some reasonable amount of time so that people can work (one reason that companies really, truly hate taking systems down for "emergency fixes" unless it's truly an emergency -- and not a manufactured emergency, either).
Even if a vendor could create the equivalent of The Security Patch That Ate Cleveland (e.g., a patch that includes fixes for all security vulnerabilities from the beginning of time), the amount of work for a customer to actually apply The Security Patch That Ate Cleveland is equivalent to upgrading to a new product version (containing all those fixes already), which many customers do regularly for other reasons. Living with constraints mean that as a vendor, you sit around and try, within some basic principles, to figure out how to do the most effective good for the most people. It doesn't mean doing the minimum and hoping nobody notices, but it does mean weighing a lot of different factors in trying to make the best use of time and people to protect the most customers you can in the most cost effective way for them.
Small digression: this is a good time to give a quick recap of the amount of work that does go into fixing security issues, which is why we continue to work to avoid these problems in the first place (through coding standards, better vulnerability testing tools, process improvements, and so on). So, here goes: Oracle has multiple large product stacks, each of which has multiple supported versions, each of which runs on multiple operating systems (the last major release of the Oracle database alone shipped on something like 19 OSs). The stacks are interdependent (for example, Oracle eBusiness Suite runs on Oracle Application Server and the Oracle Database). And of course, we have made many product acquisitions that also have "other Oracle product" dependencies.
In short, it's not enough just to fix a product problem where it occurs, you need to make sure it does not break something else that depends on the product, otherwise the "fix" is useless to a customer. And of course, all the moving parts (patches) across the company have to come out on the same day, because you don't want customers bringing down, for example, Oracle Application Server one week to apply an Oracle Database (client-side) patch, the second week to fix an Oracle Application Server bug, and the third week to patch some Oracle Collaboration Suite components that run on the middle tier. The interdependencies, time constraint (we have a fixed delivery date that can't move) and "ripple" effect are the main reason why fixing a security issue is something measured on a calendar, not a stopwatch. I try to explain this in terms of the well-known Mastercard ads:
- Two line code change fixing security bug -- 20 minutes
- Finding all similar/related bugs, on all affected product versions, fixing them, thoroughly testing the fix across all versions and product dependencies -- 3 months
- Handing customers a fix that doesn't break anything else --priceless
Where does this leave us? With a speed bump that says, in effect, newer versions of products -- almost any vendor's products -- are probably, all other things being equal, "more secure." This seems obvious, but it is worth stating. Vendors -- most of us -- know more about secure development and secure coding than we did even three or four years ago. Newer products reflect that. Also, even if we can't fix every single security issue on old product versions, we certainly are going to fix it in new versions. Preferably, as soon as we can because it is just good business and common sense to do this.
I think I should pause now and comment on a predictable screaming point -- the idea unfortunately -- but widely -- promulgated that all security issues should be fixed at exactly the same time for everybody. If they aren't, the conventional wisdom goes, the vendor is being evil-minded towards their customers.
With all due respect, that is a lot of hooey, for reasons that should be obvious.
Suppose I am a developer building a housing development containing 100 houses. Suppose also that 20 houses into my development, I realize that I have a problem with leaky bathroom sinks. In fact, it's systemic enough that I need a different sink to be dropped in. I can't just fix the leaks -- I need to swap out the sink. I have several choices here:
n I can finish the 80 other houses with the old leaky sinks, so everybody's house is equally leaky. Then, I rip out 100 sinks and retrofit them all at the same time. In the construction business (and I know this because I used to work in construction) this option is called "How Dumb Can You Be?"
n I can use the new sinks in the next house I build (number 21) and in the rest of them (houses 22-100). I can then go back and fix the 20 houses with the old bad sinks. We can argue about the exact timing of who, in houses 1 through 20, gets the new sinks when, but one thing that is clear -- if I am just about to drop a bathroom sink into house number 21, and I have a chance to put a WORKING sink in that doesn't leak, that's what I should do. Having 100 flooded bathrooms to prove a principle of equality is a recipe for being out of business. It's also really dumb, expensively so, for everybody.
Note that I am not disputing that maybe, the sink design should have been reviewed earlier, or more carefully. Or that the contractor should learn from the sink problem so new housing developments have better sinks (maybe the architect was at fault, or maybe the contractor had a bad supplier). These are all good points, and valid ones. But the reality is, and I also know this from working in construction, there is just no such thing as a building that goes up with absolutely no change orders (contract modifications due to something needing to be fixed during construction). I spent about 80% of my first job in the Navy negotiating change orders to construction contracts, and about 20% of my time managing the actual contracts. And I had mostly good architects, good engineers, and good contractors.
Oracle has tried to optimize fixing critical -- and we mean critical -- security issues in reasonably consumable chunks, four times a year for people on older product versions. We call these "critical patch updates." I also note that we actually fix security issues going forward (in new versions and patch sets) FIRST. This means, all things being equal, newer versions and patch sets have more security fixes, and generally have them sooner. We do this because, if we have a product train leaving the station, and there is a critical security problem, it makes sense -- and protects customers best -- if we get that critical issue fixed going forward if we possibly can. We bundle many -- but not all -- security fixes into critical patch updates and release those four times a year. Because of the amount of overhead in fixing, backporting, testing and integrating combined fixes, it means that there may be and often is a time lag to get a fix into older product versions (which is when we announce the fix -- when a patch goes out for those older versions). Just like the folks in houses 1-20 might have to wait to get new sinks (while people in houses 21-100 don't have that problem). It's not a perfect solution, but it is better than not fixing anything going forward to the point that everybody ends up having to install the The Security Patch That Ate Cleveland.
The most unpleasant conversations I ever have with customers, and I have had several of these, occurs when a customer has never applied any security fixes (in the days when we did security alerts) and is running on a version that was (at that time) long out-of-support. (Even with new support models we've put in place, we do not issue security alerts or CPUs for all versions.) The customer is now running on what can only be construed as an archeological version of product (as in, Oracle7 or Oracle 7.2, and I am not making that up), and yet it is a mission critical application. The customer wants to know if they are "at risk" from unpatched security vulnerabilities. I think I can safely say that they are. And I have. I also tell them that we do not do security analysis on out-of-support versions of product, but in many cases an issue we are fixing (via a critical patch update) has been in code awhile and probably does exist in the really old, out-of-support product version.
I know people like nice, stable versions of product; who doesn't? (I drove a Honda CRX for 17 years and only got a new car since the Honda was coming up on 300,000 miles and I couldn't retrofit cup holders into it.) But I tell customers they need to plan on some regular maintenance, and -- all things being equal -- newer versions of product are more secure. If I had to put this into rote form, it would be: "Dear customer: it is in your interests to upgrade from time to time, because we cannot fix every single security issue of every single severity on every single old version. Nobody can. We try as best we can to protect the most customers to the best of our ability. Part of that also means making newer versions better. Please don't move to the second-from-oldest supported version to 'get current,' please move to the latest and greatest product version if you possibly can."
I offer the above as my own speed bump -- a chance for people racing along the highway of security and in particular, security vulnerability handling to stop, look, observe, and slow down.
For more information:
Jaime Chanaga's good advice on security leadership (July 16 blog entry):
The Hailey, Idaho (former) police chief's blog on Vengeful Deer, Jesus and Bob and Big Water:
Pictures of the Trail Creek fire:
The World Center for Birds of Prey:
The release of the new Idaho quarter (with a peregrine falcon on it!):