Mary Ann Davidson Blog

In an earlier blog entry, I mentioned that I have a Siberian Husky named Thunder. I will be the first to tell you that, while I love every inch of his fat furry self, Siberians are a very demanding breed, one that is not for everybody. Siberians are very intelligent dogs, and independent to the point of willfulness (categorized as "I'm doing what I want, regardless of what you tell me to do"). I have a friend (Ira Winkler), who is a well-known security expert, and when I run into him at security conferences, we never talk about security. We talk about our Siberians (Thunder -- mine, and Bandit -- his). How clever they are, how smart they are, how much they keep us on our toes. As he says, "You want a challenge, get a Husky."

Many Siberian traits that can be irritating were actually bred into them and are very useful ones. Huskies were imported into Alaska from Siberia, where they were used by the Chukchi people -- hence, "Husky" -- as sled dogs. As such, their willfulness is good trait: Huskies are so attuned to trail conditions that they can detect drum ice,* melting ice, or other hazardous trail conditions far better than mushers can, and they will not proceed under unsafe conditions regardless of the musher's commands. Particularly in arctic terrain, you want your sled dog team to be smarter than you are: they are the "subject matter experts" on a trail. In a blizzard, you can trust them to find the trail and lead you home.

I have friends who have nice placid dogs that obey every voice command, who can be walked off lead, who roll over and act all kissy kissy licky licky, and who never challenge them. I don't have a dog like that. As much as I could occasionally strangle Thunder (such as the way he starts bugling for treats during the last 10 minutes of a two hour movie -- how does he know?), I appreciate him the way he is and wouldn't change a thing about him. In a strange way, he has helped me appreciate the various personalities I work with, (and be more accepting of my own " 'tude").

One of my direct reports, John, I hired for many reasons. He is smart, hard-working, passionate about what he does, and also "knows the trail" more than I do in many circumstances. The fact that he fights for his ideas and tells me when and where I can get off (professionally and politely, but firmly, of course), are all really great attributes. In fact, I transferred him to work for me a second time precisely because of those attributes. I did not want a "yes person" who had exactly my strengths and weaknesses and who agreed with me on everything.

I am sure Golden Retrievers are great dogs, but they are far more "aim to please" than a Husky and I would not want one leading me over the Norton Sound. Similarly, the main reason I wanted John to run security program management is that he is going to run that team based on what he sees, and not on what I think (particularly when I am giving bad directions).

Another Siberian Husky characteristic is "forging." You put a leash on your Siberian, and he leans into the harness and pulls for all his might. I'm not a small woman (135 pounds) and Thunder is not an especially big dog (155 pounds), but my dog walks me, and not the other way around. Thunder didn't need any instruction to learn to "skijor" with me, either: as soon as I put a harness on him and clipped him into my belt, he pulled me on Nordic skis. (He's unbeaten in 5Kor less and refuses to be passed. He's beaten Olympians. I don't "do" anything to get him to run. He just does. I don't "do" anything to get him to accelerate when someone is trying to pass him: he just does. Thunder loves to skijor and hates to lose.)

One of my employees has work attributes that could be called forging. She is not the supreme technician in my team (e.g., the best hacker). What Tami does have is the most determined "stick-to-it-iveness" of anybody I know. I can give her any project and, if she doesn't immediately know how to do it, she will figure it out. If she has a weakness, she will figure out how to "bone up on" that area and turn it into a strength. Tami works in the security vulnerability handling team, on multiple projects related to getting our critical patch updates out. When we asked her to do project management (keeping track of the myriad of things and people and milestones and deliverables) for the critical patch updates overall, project management was not her strong suit. It is now.

Only those who don't know her look at her and think (for example), "She could never run a marathon, or drive a sled in the Iditarod." I know she could. She'd figure out how to do it, and she would finish the race, and that much faster the next time. That attribute of "I won't quit; I won't give up, I will keep going" is remarkably rare and oh, so valuable.

These two individuals are not the only members of my team, nor the only valuable members of my team. (I look forward to bragging on more of them in future blogs.) All of them are treasures, and all of them together make for a team that is unbeatable (the "driver," so to speak, has some weak points: the team doesn't).

In making the above analogies, some people (especially those in human resources) may be appalled that I am comparing multiple excellent employees to -- sled dogs. It's a good time to recount a story that shows how heroic Siberians are, and why I think it is an apt analogy to compare my wonderful, stellar, skilled and superlative employees to Siberian Huskies. Both have wonderful attributes and have done amazing things that are largely unheralded, or are unknown or forgotten.

Many people have heard of the Iditarod, which (roughly) traces the route from Anchorage, Alaska to Nome followed in 1925 by sled dog teams relaying serum to Nome, Alaska to combat a diphtheria epidemic. Most have a two-line knowledge of this event. Three lines, if you ever saw the animated movie Balto. (Note: the dog that did most of the work was Togo, not Balto. Togo did the longest leg of the diphtheria run by far -- 261 miles, almost double what anybody else did -- including crossing Norton Sound. Many Siberian Huskies today trace their ancestry to Togo, including mine.)

And now, for the rest of the story. The city of Nome, Alaska, was blocked from sea access for much of the year -- between six and nine months. Diphtheria is a particularly horrible disease which most of us have no experience with (and should be grateful that we don't). You choke to death on your own mucus and the scaly sloughing of skin in your throat -- horribly, and painfully. When a diphtheria epidemic broke out in Nome in 1925, the city was faced with the deaths of many people -- horribly -- unless a way could be found to get diphtheria serum to them. Planes were out of the question (they could not fly at the cold temperatures and altitudes required to get to Nome). No ships could get through the ice. The only way to get the serum to Nome was by dogsled.

A man named Leonhard Sepphala organized a relay: dog teams (almost all of which were Siberians) and drivers did "legs" of the journey to race the serum to Nome. Siberians are well suited to the cold, with an outer coat that sheds snow and rain, and an inner coat that insulates them. However, the cold snap during the serum run was minus 50 degrees Fahrenheit with a wind chill, more than even a Husky can endure. Many animals ran their hearts out in the cold and died in their traces. A number could never run again. Both dogs and mushers embodied the following: "I have fought the good fight, I have finished the race, I have kept the faith." (2 Timothy 4:7) Together, they broke all records for the Nenana to Nome run.

An entire city of people owed their lives and survival to "a bunch of dogs." Their heroism is commemorated each year when the Iditarod is run, but few today really remember what those dogs and their mushers did. Every time I look into Thunder's blue and brown eyes (another breed characteristic -- eyes of different colors), I am reminded of his ancestry, and how all those quirky little things that can be -- on occasion -- so irritating, saved the lives of the people who depended on "Husky traits" in the inhospitable Yukon.

Similarly, my security team (of which I've profiled only a few people) does amazing, heroic things in (sometimes) miserable conditions, with unbelievably positive outcomes that few know about. For obvious reasons, we don't issue press releases describing the nasty security vulnerability in module X we found, or the mess that was averted because of the ideas and passion and dedication of my security team.

Furthermore, so much of the work we do is really "blazing a trail" for others. Testing new vulnerability detection tools, figuring out how to deliver secure coding training across the entire global development organization. Working with others in industry so things we "figured out" about assurance get passed to others. If you can teach an old dog new tricks -- and you can -- you can teach developers new development techniques that remove, eliminate or shut down attack vectors.

I wish there were a really good way to commemorate the "miracles every day" that my team does. There is, of course, a Hawaiian word that describes them. A Hawaiian idiom list includes all the following as part of the definition of kupaianaha: Amazing! Fantastic! Wonderful! Phenomenal! Surprising! Marvelous! Extraordinary! Astonishing! My team is truly Kupaianaha!

* Drum ice occurs when water recedes under ice. On one occasion, a musher fell through drum ice, into a large pit (where water from a stream had receded). He was unable to climb up the sides of the pit to reach the surface. All he could think to do was to tell his lead dog to go fetch a prospector who lived about an hour away, and who he had visited with his team only once before. After he sent the team away, he tried in vain to climb out of the pit, figuring the team would never be back. To his great surprise, his team reappeared a couple of hours later with the prospector. His lead dog -- a Siberian -- had understood him perfectly and done what he asked. Siberian Huskies, too are kupaianaha!

For more information:

For more on Siberian Huskies and the diphtheria serum run to Nome, read The Cruelest Miles (Thunder has an autographed copy!) by Laney and Gayle Salisbury.

More on language: the command "mush" to sled teams is from the French marchons (let's go!). Some people think racing Siberians is cruel. Try stopping them. Mine howls to go Nordic skiing with me and goes into a decline if I leave him at home.

More on Hawaiian idiom: http://www.geocities.com/TheTropics/Shores/6794/v-idioms.html

More on the Iditarod: http://www.lucidcafe.com/library/iditarod.html

More on skijoring: http://www.sleddogcentral.com/skijoring.htm

More on Leonhard Sepphala: http://en.wikipedia.org/wiki/Leonhard_Seppala

More on Togo:
http://en.wikipedia.org/wiki/Togo_%28dog%29

I also mentioned in an earlier blog that I taught my Siberian Husky Hawaiian. A useful phrase for anyone with a dog is: 

E <Hekili>, he 'ilio akamai 'oe! He 'ilio nohea/nani 'oe!  (O <Thunder>, you are a clever dog! You are a handsome/beautiful dog!) Dogs understand flattery and love in any language.

Ketchum, Idaho made the international news a few weeks ago. It seems that a Ketchum man's Burmese python, Houdini, swallowed a queen-sized electric blanket, cables and control box included, and needed surgery. I was interested in this story for several reasons. One, it's not every day that a Burmese python swallows a queen-sized electric blanket (and why queen size? You'd think the twin would be plenty filling). Second, the veterinarian who performed the surgery on Houdini, Karsten Fostvedt, happens to be my dog's vet.Thunder actually bit Karsten once when he was a puppy (but Karsten holds no grudges, thank heavens). Karsten is a great vet and even makes house calls, as he did once when I couldn't get my very recalcitrant dog to go down the hill for his annual checkup (who knew that dogs could actually grip stainless steel?) Third, it was interesting to me that this story got picked up by so many media outlets. Karsten was even on Good Morning America, and -- believe me --network news coverage does not come often to Ketchum.

A last interesting point about this was that it isn't as if Karsten really knew how to operate on a Burmese python. It probably wasn't on the final exam in veterinary school. Apparently, he made a few phone calls to consult with other vets (about where to make the incision) and then just got on with it. Houdini has an 18-inch incision and that will probably leave a similarly sized scar, but I guess if you are 12 feet long, maybe it isn't so noticeable. It's all relative.

Several years ago, I recall having a conversation with a venture capitalist (VC) about which security companies were useful and which were not. I suspect the conversation was engendered by a presentation I did (and still do) about the need for secure coding. On one slide, I stated that venture capitalists generally don't want to solve the right problem, in part because they make a lot more money on Band-Aids than vaccines. Think about it, why do we need all these "protection" products like anti-spam, anti-virus, specialty firewalls and so on? Yes, defense-in-depth is sound defensive security practice, but in general, if enterprise software were more robust, self-defending and didn't have so many dumb (technical term) coding errors (DCEs), we wouldn't need so many products that are supposed to protect against attacks engendered by DCEs.

You can keep selling anti-virus as long as viruses are so easy to write and distribute and products are so vulnerable to them. You can also keep selling the latest "cure-cancer-raise-the-dead-and-protect-against-all-attacks-known-to-mankind" product, and by doing so, you can print money. Printing money makes investors -- like VCs -- very happy. If products were actually much stronger innately (because they were well-designed and free or largely free of coding errors that enable attacks), and had a notion of self-defense, the Band-Aid companies would be out of business or their market wouldn't be so large.

Ultimately, Band-Aids don't actually cure diseases, anyway. They just protect a wound and give it time to heal. They don't do anything for a serious problem, though. Just like Houdini wasn't going to be able to take two aspirin to cure his stomach upset; he needed major surgery.

Similarly, for products that need security patches -- and most of them do from time to time -- customers ultimately have no choice but to apply the patches. Vendors have to fix the problem, anyway (do code surgery) and patients need to follow the doctor's advice (apply the patches), anyway. We know from process manufacturing that the earlier you fix a defect, the less expensive it is to do so. The more bugs you find during development and fix, the less costly it is for vendors and customers to maintain the products. Back to the medical example, you'd ideally like to get rid of problems while they are small (like dealing with a niggling cough) before they become a much larger and more widespread problem (e.g., pneumonia).

One of the right problems to solve, I said to Ted, the VC with whom I had the conversation, was the secure coding problem. Vendors need better tools to find defects in software during development, so they can find and fix more problems before products ship. Much to my amazement, Ted said that he was actually tired of funding companies that solved the wrong problem, and to that end, he was funding a startup called Fortify Software that was building a tool to do static analysis of code.

Static analysis tools look through your source code to find coding errors -- specifically, exploitable coding errors -- that can create problems later. The good ones do data flow analysis, looking at how input actually flows through an application, and thus they can determine how and whether "bad input" can do actual damage to your application. "Exploitable" is the key word here, because, while coding errors are coding errors, tools that raise lots of alarms for "looks like a problem" make you spend eons chasing down "looks like a problem" alarms which may turn out not to be problems at all. False positives are the bane of many code scanning tools. It's not just that tracking them down is such a waste of time, it's that developers, once harassed 9 times for the 10^th item that turns out to be a real problem, are not going to ever want to touch the tool again. A good tool helps developers to do the right thing, motivates them to do the right thing, and helps embed a culture of security in your development environment. A bad one makes developers leery of secure coding programs. And of course, you want a tool that doesn't miss a Big Honking Hole (BHH), either, because that could be the one that your not-so-friendly neighborhood hacker sneaks through. (Note: The industry-standard term for "missing BHHs in a scan is "false negatives.")

Rooting out bad code is not dissimilar from trying to get an electric blanket out of a big snake. It's completely embedded, you know the snake is going to do a lot better without all the bad stuff in there, and it's not as if this is all that easy to do. You probably need to consult with experts. (In the case of Oracle, even if every single developer were an expert, trying to go through 50 million lines of code by hand is a non-starter).

I also have to add that every time I would give that talk about the need for secure coding, every "code scanning" company in the world would come out of the woodwork and extol the virtues of their product. I actually had an email template I'd send off to these well-meaning hucksters (OK, well, some were hucksters, some had probably an OK product that ultimately was not going to work for Oracle). We needed a product that could handle millions of lines of code without choking (another technical term), would work in our development environment (e.g., with our tools, and on our development operating system and in our server farms), had a really low rate of false positives, could be extensible by us to add coding rules we wanted to write - and more. Usually, what these companies had was something that had never been tested on millions of lines of code, didn't scale, raised lots of bogus alarms, wasn't extensible (unless you hired their very expensive consultants) and wouldn't handle all the languages we had (C, Java and PL/SQL -- much less the interfaces between them). Oh, and they only worked on an operating system that wasn't the one we develop on. Sometimes, they wanted you to complete revamp your development process just to use their tool (kind of like having to redecorate your entire house when all you wanted was to get rid of termites).  Even when I pointed out that their products didn't meet any of our criteria,  "Other than that, it's a perfect fit!" their sales guys would say confidently. ("Perfect fit" like a size 12 shoehorned into a size 6 bikini: it's not pretty, and it sure isn't comfortable.)

Over a period of time, during which Oracle was vetting many tools to help root out bad code, Fortify Software contacted us at Ted's suggestion. I have to say in all honesty that Fortify Software didn't pass all our tests at first, either. But they kept coming back. They kept working with us to handle what we needed. They did not give up (of course, neither did we). They understood innately that no matter how cool a static analysis tool is technically, it has to work in your development environment and with your development processes or it's just going to end up as expensive shelfware.

Some of you may have read that we licensed Fortify Software last fall for use across our technical stack products. As an aside, sometimes licensing deals include things like vendors' willingness to say nice things about the product. In this case, I am pretty keen to say nice things about Fortify Software without prompting because I am so grateful to have found something that solves the right problem -- or one of them -- works for us, and is helping us find coding errors earlier and earlier in the development process. What's not to like?  I am not, in general, a technophile but I am almost giddy about Fortify Software. It's right up there with my 4/3 "Psycho" wetsuit in terms of affection for technical marvels (it's got all taped seams, is ultraflexible, fits perfectly and I stay really toasty down to 49 degree Fahrenheit water).  Given how I feel about surfing, this is high praise, indeed.

We use other tools to root out bad code, some of which we developed ourselves in addition to licensing Fortify Software, and all of which I will talk about in another blog.

I like bragging about (well, maybe "complimenting" is a more business-like term) other people who work in security and who help make us successful. A key individual here is one of the release management kahunas for our technical stack products (database, application server, identity management, and so on): Mark. My team did the initial vetting of Fortify Software and has contributed some of the aforementioned home-grown tools to the secure coding cause, but Mark was the person who worked day in and out with Fortify Software and laid out the changes he wanted in the product to work best in our development environment. He's also the one responsible for rolling this into development processes (e.g., to include adding time to the release management schedule to fix what we find). He's also the "enforcer" who ensures that developers have taken our secure coding class. Mark is one of the nicest guys on planet earth and without his embedding security into development and making it stick, my team wouldn't be nearly as effective. Mark speaks at a number of conferences on things like how you vet "code scanning tools" and how you embed them into your release processes (and he talks about neat little tricks we did ourselves, such as the way we turned 250,000 regression tests into 250,000 destructive security tests). Mahalo nui loa, Mark.

One of the oddities one sees here in Ketchum in the height of summer is- Nordic skiing. No, I am not making that up. We have bike trails in the summer (that are Nordic trails in winter) and along those bike trails, in addition to cyclists, joggers and dog walkers, we have people on "roller-Nordic skis" practicing their sport for the upcoming winter season. (Nordic and alpine skiing-- and rodeo -- are all varsity sports in the high schools here.) Both traditional Nordic and skate-skiing roller skis are available. I am feeling a bit intimidated, as I had planned on doing the Boulder Mountain Tour (32K) next year and am feeling out of shape for it, more so as I see all these folks blitzing along the Nordic trails in 85 degree weather. The bottom line is that all these hard core Nordic skiers are practicing and training almost daily to perfect their technique. Lack of snow is an impediment but not an obstacle to the pursuit of Nordic excellence.

Secure coding is like that, and to that end, "daily practice" is kind of important for people to instill good habits and "perfect their sport." One of the enhancements Fortify Software is making for Oracle is the ability to run static analysis against a single transaction, at developer request, so that a developer can get immediate feedback and fix the issue before even checking the transaction into source control. Sure, running Fortify Software weekly or daily finds plenty of DCEs, but how much better to give the developer instantaneous feedback so he/she can fix the problem before even checking in the code? (And I mean "feedback": things like pointing out the exact line in code of the problem, what it is, why it matters, and links to descriptions of "what is a buffer overflow, anyway, and why should you care?") The scan results can be stored centrally and we can thus have metrics on what's found and fixed before product ships. It also helps prevent test result suppression, as if anybody would ever do that in a rush to get his or her code checked in (!)

Most of the people in Ketchum who Nordic ski have done so for years, and many have done the Boulder Mountain Tour for years, as well. My first landlord in Ketchum just turned 70 and she has won her age division in the Boulder Mountain Tour for the last few years. (Go, Shauna!) The point is, these sportsmen and sportswomen are in this for the long haul -- not just to do the Boulder Mountain Tour but also to do it year after year. Just because you are older doesn't mean you are not competitive (my goal is to be an old lady who still rips on a short board).

Similarly, Oracle has been -- and is -- in security for the long haul, and just as daily training is part of being able to ski the Boulder Mountain Tour year after year -- being able to check our code daily is important to be able to turn out secure code -- day after day, year after year. Thanks to people like Ted, the folks at Fortify Software, and Mark, I expect us to continue to code securely for years to come. In fact, I want our code to rip.

For more on Fortify Software:

www.fortifysoftware.com

For more on Houdini and the electric blanket:

http://www.usatoday.com/news/offbeat/2006-07-19-python_x.htm

http://news.bbc.co.uk/2/hi/americas/5197042.stm

For more on the Boulder Mountain Tour:

http://www.bouldermountaintour.com/

For more on my absolutely stupendous Psycho II wetsuit:

http://www.oneill.com/juniors-products.php?sport=surf&categoryID=-1&typeID=20&seriesID=40&id=884

Before I get started, it's a good time to correct errata that occurred in previous blog entries.

First of all, my dog is not 155 pounds--which should have read "55 pounds," anyway--but is, according to the scale at my vet's, 67 pounds. And why you care is because of my comment that my dog walks me and not the other way around. If he did outweigh me by 20 pounds, as originally reported, it's not so unusual that he can pull me around. (Though I defy anyone to maintain footing when Thunder sees--as he did last winter--11 uninvited elk bedded down in his backyard.  Big excitement. He pulled me right over.)

Second of all, I am pleased to report that Jack Love, my former Hebrew teacher, whom I had reported as no longer teaching Hebrew is, in fact, still teaching Hebrew. Barukh ha-Shem (Hebrew for "Praise the Lord," though literally, "Bless the Name!") I hope to see him at Oracle OpenWorld and catch up on his family and the Qal conjugation. He's about to start teaching Aramaic, which is also a cool language.

*************

I am writing this in advance of Labor Day. The welcome prospect of a three-day weekend got me to thinking about how we work, why we work, and how almost everything in our lives is now geared to cramming in as much activity as possible to the allotted time. (And yes, ultimately there is a security issue there.) The way so many of us work, you almost think we are hoping our epitaph will read: "My inbox was empty," which would be sad, indeed.

The reality is that you need a certain amount of time to do certain activities well, and shortchanging the time means the activity is shortchanged, also. Nothing very profound there. Whether you are talking about work or leisure, time isn't unlimited and the "when" of time is often as important as the "how much" of time.

For example, when I am in San Francisco, I like to surf after work. Generally, the later I go, the better, since I enjoy catching a nice sunset and the beach isn't as crowded close to dark. (One of the guys at my local surf break is convinced I am a vampire, since he never sees me in daylight.)  The only problem is that, when I don't pay attention to when sunset actually is (it's published in the paper every day, generally with the tide tables that are a surfer's friend), or I get caught in traffic getting to the beach, I have a really small window of time in which to surf.

I might be--and often am--very motivated to catch lots of waves after a hard day, but no matter how hard I push myself, you only get 20 minutes or so of "afterglow" after the sun goes down. Ceteris paribus (fancy Latin term economists use that means "all things being equal") I just cannot catch as many waves in 20 minutes as I can when I'm out in the lineup for, say, 2 hours. Surfing in darkness is risky both because you have to feel--not see--the swell, and because it is "predator feeding time." Shudder.

A subtler point is that the joy of surfing isn't merely the number of waves I catch. If I miss the sunset (because I had to do "just one more email"), it isn't just that I get fewer waves because there is only a small window of daylight. I missed the ineffable beauty of watching the sun go down from my surfboard, and I will never get it back. Carpe diem ("seize the day"), indeed.

Time is always a factor in product development cycles. You have a certain amount of overhead related to just getting your product out. On the front end, this can include requirements gathering, design, test plans and reviews to make sure you got the requirements and design nailed before you start actual coding. On the back end, testing, documentation, training (your sales folks and support analysts need to understand the product), and for Oracle, rolling products out in production, since we run our own company on our own software. "Eating our own dogfood," is the industry-standard phrase, though I prefer "sipping our own champagne." The actual coding time isn't always that long compared to the overhead of getting the release out ("overhead" here is not meant pejoratively). In some cases, it doesn't need to be, just as baking a pie goes a lot faster if you've laid out all the ingredients, measuring cups, utensils, and preheated the oven ahead of time. (I can make a rhubarb pie in about 15 minutes if I use pre-made piecrusts and have everything ready to cut and measure before starting.)

I joke from time to time that the software industry is the only one that tries to shoehorn in extra things to a product on the grounds that they haven't actually cut the tape (or burned a CD) yet. Imagine what would happen to automobiles if, say, General Motors decided a week before cars were to be manufactured that they should add an extra row of seats and throw the air-conditioner in the trunk. Needless to say, you don't always get good results when you make drastic changes late in the game. (Like deciding to make a peach pie instead of a rhubarb pie just before I pop dessert in the oven. Uh, it's too late, really. The crust is already sealed.) Development process is important, and so is the actual discipline of adhering to the process. I have to discipline myself to leave work at a reasonable hour if I want to improve my surfing (through regular practice) and people building software have to discipline themselves not to add a ton of stuff 20 minutes before the product ships (or suffer the consequences).

Sometimes there is a tendency to think that more resources will help a time crunch. The reality is that at some point you cannot successfully add more people or "machines" to get something faster. In my earlier example, more surfboards wouldn't help me catch more waves in 20 minutes. Similarly, 9 women can't collectively have a baby in one month. I took a quantitative methods class in business school that impressed this lesson upon me very well. (In quantitative methods, you had problems you were supposed to optimize the solutions for and yes, we used computers to do it, and yes, that's when it first dawned on me that I was really a lousy programmer.)  Probably the most important lesson I took away from that class was that (and of course, the programming language we used also imposed this limitation): you simply can't optimize on all parameters. You can't maximize A, B and C and minimize D, E and F. A glib way of remembering this is the expression: "You can have it right, or you can have it now, but you can't always have it right now."

Where does this leave us as regards security? First of all, while people sometimes complain about process "adding time to releases," good processes are developers' friends. I once worked on a vertical market product team where we did't "own the code." The team that did "own the code" we were adding a feature to was quite proprietary about it (even insisting we indent their code their way), including rigorous reviews of functional and design specs. Wonder of wonders, due to all the reviews and "work upfront" to make sure we knew exactly how our new accounting feature was supposed to work and everybody bought off on it, there were no changes to the feature once the developer started coding. He just sailed right along, coded it in record time and the feature did exactly what it was supposed to do--and no more. It was one of the smoothest projects I have ever worked on and made me a process fanatic. The result was high quality code, and few bugs, since he knew what he was going to code before he started and there were no late-breaking changes to add complexity and "unintended consequences."

Thrashing around because you forgot something or had to go back to redo something all wastes time. Good process doesn't produce perfect product, or perfectly secure product, but more people won't fix a bad process. Developers would rather "code smoothly" than redo things, which is why (beating this horse to an efficient death), we really try to embed security into the development process itself and have a strong development process that we stick to with discipline. If you haven't discovered this yourself, "just start coding" is not a development process.

Second, and perhaps more subtle point: you need to encourage good behavior as regards secure coding practice, to include measuring and rewarding the right things--and giving people enough time to do what you are expecting them to do. I've (on rare occasions) had developers sidle up to me and say that they wanted to take the time to do all the correct input validation, but their manager was more concerned about getting the feature done than doing the right validation. True, nobody has infinite time to work on a release. True, we do want to add new cool enhancements. More true, it's "pay now or pay later" as regards secure coding. You code it right the first time or I can abso-bloody-lutely guarantee you will be revisiting that code later. I talked in an earlier blog about the importance of adding time to a development cycle (once you start using automated tools), so you give developers time to fix the problems they find in their code.  You want people to use the tools, and fix what they find. Encourage and reward good behavior by allowing time for people to do good work.

One of the changes we've implemented to try to reward the right behavior is (with regard to bugs), while the general metric once you do find a bug is "how fast did you close it?," the relevant metric with regard to security bugs is "did this bug ever get reopened?" You can get someone to close a security bug really fast if they only fix the exact manifestation or fault that was reported.  What you want is for them to fix it completely, which--ceteris paribus--may mean that that it takes longer to fix it, to find and fix all conditions that will cause you problems, not just the single one that was reported. You can have it right, or you can have it now, but you can't always have it right now.

One last point about time, which perhaps is what I was leading up to, is that sometimes the best thing you can do with it is respect it and respect people's use of it. If you don't, you lose good people, which isn't a security problem per se but it is a management one.

When I was in the Navy, I was literally on duty 24 hours a day. Many military officers give up part of their weekends regularly to stand watch (I am proud to say, we never had enemies of the United States break into my barracks building in Newport, Rhode Island. Everyone at the Naval Education and Training Center could sleep soundly on that point.)  Also, you are never not a naval officer. Your time is the Navy's because you are always on duty, to a point. You can have leave (vacation) cancelled for things like, oh, war breaking out. Most people in the armed forces understand things like "I am sorry, your honeymoon is cancelled, the ship has to deploy to quell unrest in (name troubled spot in the world)."

For the most part, corporate America (or corporate anywhere) does not have "national emergencies" like that. We do have customers who operate around the world, and many companies have adapted their business (e.g., their support organizations) to "follow the customers around the globe." This is just being a good supplier and partner of customers. That's a different thing from asking a particular individual to "follow the globe" and stay up for 24 hours. People who work in IT like machines that are up 7 by 24, but realistically, people need regular "downtime" or they have a "core dump." Unfortunately, some of us suffer from a lack of perspective that treats time as something that we always have plenty of, and that means we are not willing to make reasonable and considerate choices in terms of how we spend our time and other people's time. Employees are generally not on duty 24 hours a day, and shouldn't be.

A long time ago, I worked for a manager who (I am not making this up) asked my coworker not to go on her honeymoon for something work-related. Mind you, she and her husband had planned their honeymoon around the Olympics that year (and had gotten tickets) so postponing the honeymoon really meant derailing their plans for 4 years, not a couple of weeks. I am pleased to note that my then-manager's Senior Vice President got wind of the "postpone your honeymoon" VP's management techniques, ran down to the VP's office, closed the door and said "What are you thinking? Let the woman go on her honeymoon!" Nobody remembers what was allegedly so critical that my coworker should have given up her honeymoon, but I can guarantee you that had that VP persisted, I would have lost a valuable coworker (and her husband, who also worked for the company).

One way to gauge how important something is that you are asking an employee to give up something personal for it is to ask the question: "Will this matter in a week? A month? A year?" Time is so precious and (generally speaking), you are only going to have one honeymoon. Maybe there is a sunset every evening, but oh, the beautiful ones I have seen-- each one of them different, and I treasure them all.

 The origin of the word "Sabbath" is the Hebrew word shabbat (ת×�שׂ). Shabbat means "cease" or "rest," and the first place it appears in the Bible is the book of Genesis (Bereshith, in Hebrew).  "And on the seventh day God ended his work which he had made; and he rested on the seventh day from all his work which he had made." (Gen 2:2) Which is where we get both the idea of a Sabbath, and the word for it. From time to time in my career I have had to work on weekends, if not through the weekend. Weekends when I did naval reserve duty, for example, or times when I worked in development organizations and we were trying to ship product. "Sometimes, you just gotta." But as a general rule, I do like to remember that even God rested on the seventh day. (And thus, being human, I figure it should be OK that we take two days off every week.)

Many people are familiar with a section of the book of Ecclesiastes, in no small part because it was made into a hit song in the 1960s called "Turn, Turn, Turn." The passage begins, "There is a time for everything, and a season for every activity under heaven." You can listen to the song (The Byrds recorded it) or check out Ecclesiastes 3 if you want to read the entire thing. Most people miss the punchline, which is even more lyrical:

9 What does the worker gain from his toil? 10 I have seen the burden God has laid on men. 11 He has made everything beautiful in its time.

Labor Day is a good day to rest from your labors, and to experience something beautiful in its time. The work will still be there tomorrow. Me, I have an appointment with a fly rod and the Big Wood River.

Of course, now I am too old to be much of a fisherman, and now of course I usually fish the big waters alone, although some friends think I shouldn't. Like many fly fishermen in western Montana where the summer days are almost Arctic in length, I often do not start fishing until the cool of the evening. Then in the Arctic half-light of the canyon, all existence fades to a being with my soul and memories and the sounds of the Big Blackfoot River and a four-count rhythm and the hope that a fish will rise.

Eventually, all things merge into one, and a river runs through it. The river was cut by the world's great flood and runs over rocks from the basement of time. On some of the rocks are timeless raindrops. Under the rocks are the words, and some of the words are theirs. I am haunted by waters.

A River Runs Through It, by Norman MacLean

For more information:

On the meaning of ceteris paribus:<o:p></o:p>

http://en.wikipedia.org/wiki/Ceteris_paribus

An interlineal (Hebrew and English) Bible entry for Genesis 2:

http://www.blueletterbible.org/kjv/Gen/Gen002.html

And for Ecclesiastes 3:

http://www.blueletterbible.org/kjv/Ecc/Ecc003.html#top]

More about A River Runs Through It:

http://www.press.uchicago.edu/Misc/Chicago/500667.html

More about flyfishing on the Big Wood River:

http://www.flyfishingconnection.com/bigwood.html