I Can't Believe I Ate That!
By MT:15 on Aug 17, 2006
Ketchum, Idaho made the international news a few weeks ago. It seems that a Ketchum man's Burmese python, Houdini, swallowed a queen-sized electric blanket, cables and control box included, and needed surgery. I was interested in this story for several reasons. One, it's not every day that a Burmese python swallows a queen-sized electric blanket (and why queen size? You'd think the twin would be plenty filling). Second, the veterinarian who performed the surgery on Houdini, Karsten Fostvedt, happens to be my dog's vet.Thunder actually bit Karsten once when he was a puppy (but Karsten holds no grudges, thank heavens). Karsten is a great vet and even makes house calls, as he did once when I couldn't get my very recalcitrant dog to go down the hill for his annual checkup (who knew that dogs could actually grip stainless steel?) Third, it was interesting to me that this story got picked up by so many media outlets. Karsten was even on Good Morning America, and -- believe me --network news coverage does not come often to Ketchum.
A last interesting point about this was that it isn't as if Karsten really knew how to operate on a Burmese python. It probably wasn't on the final exam in veterinary school. Apparently, he made a few phone calls to consult with other vets (about where to make the incision) and then just got on with it. Houdini has an 18-inch incision and that will probably leave a similarly sized scar, but I guess if you are 12 feet long, maybe it isn't so noticeable. It's all relative.
(For those who want to know what could possess a snake to swallow an electric blanket, the theory is that the warm blanket got tangled up with Houdini's rabbit dinner and I don't want to think about that in too much detail, as I am a confirmed critter-lover and a bunny lover in particular.)
Several years ago, I recall having a conversation with a venture capitalist (VC) about which security companies were useful and which were not. I suspect the conversation was engendered by a presentation I did (and still do) about the need for secure coding. On one slide, I stated that venture capitalists generally don't want to solve the right problem, in part because they make a lot more money on Band-Aids than vaccines. Think about it, why do we need all these "protection" products like anti-spam, anti-virus, specialty firewalls and so on? Yes, defense-in-depth is sound defensive security practice, but in general, if enterprise software were more robust, self-defending and didn't have so many dumb (technical term) coding errors (DCEs), we wouldn't need so many products that are supposed to protect against attacks engendered by DCEs.
You can keep selling anti-virus as long as viruses are so easy to write and distribute and products are so vulnerable to them. You can also keep selling the latest "cure-cancer-raise-the-dead-and-protect-against-all-attacks-known-to-mankind" product, and by doing so, you can print money. Printing money makes investors -- like VCs -- very happy. If products were actually much stronger innately (because they were well-designed and free or largely free of coding errors that enable attacks), and had a notion of self-defense, the Band-Aid companies would be out of business or their market wouldn't be so large.
Ultimately, Band-Aids don't actually cure diseases, anyway. They just protect a wound and give it time to heal. They don't do anything for a serious problem, though. Just like Houdini wasn't going to be able to take two aspirin to cure his stomach upset; he needed major surgery.
Similarly, for products that need security patches -- and most of them do from time to time -- customers ultimately have no choice but to apply the patches. Vendors have to fix the problem, anyway (do code surgery) and patients need to follow the doctor's advice (apply the patches), anyway. We know from process manufacturing that the earlier you fix a defect, the less expensive it is to do so. The more bugs you find during development and fix, the less costly it is for vendors and customers to maintain the products. Back to the medical example, you'd ideally like to get rid of problems while they are small (like dealing with a niggling cough) before they become a much larger and more widespread problem (e.g., pneumonia).
One of the right problems to solve, I said to Ted, the VC with whom I had the conversation, was the secure coding problem. Vendors need better tools to find defects in software during development, so they can find and fix more problems before products ship. Much to my amazement, Ted said that he was actually tired of funding companies that solved the wrong problem, and to that end, he was funding a startup called Fortify Software that was building a tool to do static analysis of code.
Static analysis tools look through your source code to find coding errors -- specifically, exploitable coding errors -- that can create problems later. The good ones do data flow analysis, looking at how input actually flows through an application, and thus they can determine how and whether "bad input" can do actual damage to your application. "Exploitable" is the key word here, because, while coding errors are coding errors, tools that raise lots of alarms for "looks like a problem" make you spend eons chasing down "looks like a problem" alarms which may turn out not to be problems at all. False positives are the bane of many code scanning tools. It's not just that tracking them down is such a waste of time, it's that developers, once harassed 9 times for the 10th item that turns out to be a real problem, are not going to ever want to touch the tool again. A good tool helps developers to do the right thing, motivates them to do the right thing, and helps embed a culture of security in your development environment. A bad one makes developers leery of secure coding programs. And of course, you want a tool that doesn't miss a Big Honking Hole (BHH), either, because that could be the one that your not-so-friendly neighborhood hacker sneaks through. (Note: The industry-standard term for "missing BHHs in a scan is "false negatives.")
Rooting out bad code is not dissimilar from trying to get an electric blanket out of a big snake. It's completely embedded, you know the snake is going to do a lot better without all the bad stuff in there, and it's not as if this is all that easy to do. You probably need to consult with experts. (In the case of Oracle, even if every single developer were an expert, trying to go through 50 million lines of code by hand is a non-starter).
I also have to add that every time I would give that talk about the need for secure coding, every "code scanning" company in the world would come out of the woodwork and extol the virtues of their product. I actually had an email template I'd send off to these well-meaning hucksters (OK, well, some were hucksters, some had probably an OK product that ultimately was not going to work for Oracle). We needed a product that could handle millions of lines of code without choking (another technical term), would work in our development environment (e.g., with our tools, and on our development operating system and in our server farms), had a really low rate of false positives, could be extensible by us to add coding rules we wanted to write - and more. Usually, what these companies had was something that had never been tested on millions of lines of code, didn't scale, raised lots of bogus alarms, wasn't extensible (unless you hired their very expensive consultants) and wouldn't handle all the languages we had (C, Java and PL/SQL -- much less the interfaces between them). Oh, and they only worked on an operating system that wasn't the one we develop on. Sometimes, they wanted you to complete revamp your development process just to use their tool (kind of like having to redecorate your entire house when all you wanted was to get rid of termites). Even when I pointed out that their products didn't meet any of our criteria, "Other than that, it's a perfect fit!" their sales guys would say confidently. ("Perfect fit" like a size 12 shoehorned into a size 6 bikini: it's not pretty, and it sure isn't comfortable.)
Over a period of time, during which Oracle was vetting many tools to help root out bad code, Fortify Software contacted us at Ted's suggestion. I have to say in all honesty that Fortify Software didn't pass all our tests at first, either. But they kept coming back. They kept working with us to handle what we needed. They did not give up (of course, neither did we). They understood innately that no matter how cool a static analysis tool is technically, it has to work in your development environment and with your development processes or it's just going to end up as expensive shelfware.
Some of you may have read that we licensed Fortify Software last fall for use across our technical stack products. As an aside, sometimes licensing deals include things like vendors' willingness to say nice things about the product. In this case, I am pretty keen to say nice things about Fortify Software without prompting because I am so grateful to have found something that solves the right problem -- or one of them -- works for us, and is helping us find coding errors earlier and earlier in the development process. What's not to like? I am not, in general, a technophile but I am almost giddy about Fortify Software. It's right up there with my 4/3 "Psycho" wetsuit in terms of affection for technical marvels (it's got all taped seams, is ultraflexible, fits perfectly and I stay really toasty down to 49 degree Fahrenheit water). Given how I feel about surfing, this is high praise, indeed.
We use other tools to root out bad code, some of which we developed ourselves in addition to licensing Fortify Software, and all of which I will talk about in another blog.
I like bragging about (well, maybe "complimenting" is a more business-like term) other people who work in security and who help make us successful. A key individual here is one of the release management kahunas for our technical stack products (database, application server, identity management, and so on): Mark. My team did the initial vetting of Fortify Software and has contributed some of the aforementioned home-grown tools to the secure coding cause, but Mark was the person who worked day in and out with Fortify Software and laid out the changes he wanted in the product to work best in our development environment. He's also the one responsible for rolling this into development processes (e.g., to include adding time to the release management schedule to fix what we find). He's also the "enforcer" who ensures that developers have taken our secure coding class. Mark is one of the nicest guys on planet earth and without his embedding security into development and making it stick, my team wouldn't be nearly as effective. Mark speaks at a number of conferences on things like how you vet "code scanning tools" and how you embed them into your release processes (and he talks about neat little tricks we did ourselves, such as the way we turned 250,000 regression tests into 250,000 destructive security tests). Mahalo nui loa, Mark.
One of the oddities one sees here in Ketchum in the height of summer is- Nordic skiing. No, I am not making that up. We have bike trails in the summer (that are Nordic trails in winter) and along those bike trails, in addition to cyclists, joggers and dog walkers, we have people on "roller-Nordic skis" practicing their sport for the upcoming winter season. (Nordic and alpine skiing-- and rodeo -- are all varsity sports in the high schools here.) Both traditional Nordic and skate-skiing roller skis are available. I am feeling a bit intimidated, as I had planned on doing the Boulder Mountain Tour (32K) next year and am feeling out of shape for it, more so as I see all these folks blitzing along the Nordic trails in 85 degree weather. The bottom line is that all these hard core Nordic skiers are practicing and training almost daily to perfect their technique. Lack of snow is an impediment but not an obstacle to the pursuit of Nordic excellence.
Secure coding is like that, and to that end, "daily practice" is kind of important for people to instill good habits and "perfect their sport." One of the enhancements Fortify Software is making for Oracle is the ability to run static analysis against a single transaction, at developer request, so that a developer can get immediate feedback and fix the issue before even checking the transaction into source control. Sure, running Fortify Software weekly or daily finds plenty of DCEs, but how much better to give the developer instantaneous feedback so he/she can fix the problem before even checking in the code? (And I mean "feedback": things like pointing out the exact line in code of the problem, what it is, why it matters, and links to descriptions of "what is a buffer overflow, anyway, and why should you care?") The scan results can be stored centrally and we can thus have metrics on what's found and fixed before product ships. It also helps prevent test result suppression, as if anybody would ever do that in a rush to get his or her code checked in (!)
Most of the people in Ketchum who Nordic ski have done so for years, and many have done the Boulder Mountain Tour for years, as well. My first landlord in Ketchum just turned 70 and she has won her age division in the Boulder Mountain Tour for the last few years. (Go, Shauna!) The point is, these sportsmen and sportswomen are in this for the long haul -- not just to do the Boulder Mountain Tour but also to do it year after year. Just because you are older doesn't mean you are not competitive (my goal is to be an old lady who still rips on a short board).
Similarly, Oracle has been -- and is -- in security for the long haul, and just as daily training is part of being able to ski the Boulder Mountain Tour year after year -- being able to check our code daily is important to be able to turn out secure code -- day after day, year after year. Thanks to people like Ted, the folks at Fortify Software, and Mark, I expect us to continue to code securely for years to come. In fact, I want our code to rip.
For more on Fortify Software:
For more on Houdini and the electric blanket:
For more on the Boulder Mountain Tour:
For more on my absolutely stupendous Psycho II wetsuit: