Wednesday Mar 25, 2015

Is Your Shellshocked Poodle Freaked Over Heartbleed?

Security weenies will understand that the above title is not as nonsensical as it appears. Would that it were mere nonsense. Instead, I suspect more than a few will read the title and their heads will throb, either because the readers hit themselves in the head, accompanied by the multicultural equivalents of “oy vey” (I’d go with “aloha ‘ino”), or because the above expression makes them reach for the most potent over- the-counter painkiller available.

For those who missed it, there was a sea change in security vulnerabilities reporting last year involving a number of mass panics around “named” vulnerabilities in commonly-used – and widely-used – embedded libraries. For example, the POODLE vulnerability (an acronym for Padding Oracle On Downgraded Legacy Encryption) affects SSL version 3.0, and many products and services using SSL version 3.0 use third party library implementations. The Shellshock vulnerabilities affect GNU bash, a program that multiple Unix-based systems use to execute command lines and command scripts. These vulnerabilities (and others) were widely publicized (the cutesie names helped) and resulted in a lot of scrambling to find, fix, and patch the vulnerabilities. The cumulative result of a number of named vulnerabilities last year in widely-used and deployed libraries I refer to as the Great Shellshocked Poodle With Heartbleed Security Awakening (GSPWHSA). It was a collective IT community eye opener as to:

The degree to which common third party components are embedded in many products and services
The degree to which vendors (and customers) did not know where-all these components actually were used, or what versions of them were used
And, to some degree (more on which below) the actual severity of these issues

A slight digression on how we got to a Shellshocked Poodle with Heartbleed. Way back in the olden days (when I started working at Oracle), the Internet hadn’t taken off yet, and there weren’t as many standard ways of doing things. The growth of the Internet led to the growth of standards (e.g., SSL, now superseded by TLS) so Stuff Would Work Together. The requirement for standards-based interoperability fostered the growth of common libraries (many of them open source), because everyone realized it was dumb to, say, build your own pipes when you could license someone else’s ready-made pipe libraries. Open source/third party libraries helped people build things faster that worked together, because everyone wasn’t building everything from scratch. None of these – standards, common libraries, open source – are bad things. They are (mostly) very good things that have fostered the innovation we now take for granted.

Historically, development organizations didn’t always keep careful track of where all the third party libraries were used, and didn’t necessarily upgrade them regularly. To some degree, the “not upgrade” was understandable – unless there is a compelling reason to move from Old Reliable to New and Improved (as in, they actually are improved and there is a benefit to using the new stuff), you might as well stick with Old and Reliable. Or so it seemed.

When security researchers began focusing on finding vulnerabilities in widely-used libraries, everyone got a rude awakening that their library of libraries (that is, listing of what components were used where) needed to be a whole lot better, because customers wanted to know very quickly the answer to “is the product or cloud service I am using vulnerable?” Moreover, many vendors and service providers realized that, like it or not, they needed to aggressively move to incorporate reasonably current (patched) versions of libraries because, if the third party component you embed is not supported for the life of the product or service you are embedding it in, you can’t get a security patch when you need one: in short, “you are screwed,” as we security experts say. I’ve remarked a lot recently, with some grumbling, that people don’t do themselves any favors by continuing to incorporate libraries older than the tablets of Moses (at least God is still supporting those).

Like all religious revivals, the GSPWHSA has thus resulted in a lot of people repenting of their sins: “Forgive me, release manager, for I have sinned, I have incorporated an out-of-support library in my code.” “Three Hail Marys and four version upgrades, my son…” Our code is collectively more holy now, we all hope, instead of continuing to be hole-y. (Yes, that was a vile pun.) This is a good thing.

The second aspect of the GSPWHSA is more disturbing, and that is, for lack of a better phrase, the “marketing of security vulnerabilities.” Anybody who knows anything about business knows how marketing can – and often intends to – amplify reality. Really, I am sure I can lose 20 pounds and find true love and happiness if I only use the right perfume: that’s why I bought the perfume! Just to get the disclaimer out of the way, no, this is not another instance of the Big Bad Vendor complaining about someone outing security vulnerabilities. What’s disturbing to me is the outright intent to create branding around security vulnerabilities and willful attempt to create a mass panic – dare we say “trending?” – around them regardless of the objective threat posed by the issue. A good example is the FREAK vulnerability (CVE-2015-0204). The fix for FREAK was distributed by OpenSSL on January 8th. It was largely ignored until early March when it was given the name FREAK.  Now, there are a lot of people FREAKing out about this relatively low risk vulnerability while largely ignoring unauthenticated, network, remote code execution vulnerabilities.

Here’s how it works. A researcher first finds vulnerability in a widely-used library: the more widely-used, the better, since nobody cares about a vulnerability in Digital Buggy Whip version 1.0 that is, like, so two decades ago and hardly anybody uses. OpenSSL has been a popular target, because it is very widely used so you get researcher bragging rights and lots of free PR for finding another problem in it. Next, the researcher comes up with a catchy name. You get extra points for it being an acronym for the nature of the vulnerability, such as SUCKS – Security Undermining of Critical Key Systems. Then, you put up a website (more points for cute animated creature dancing around and singing the SUCKS song). Add links so visitors can Order the T-shirt, Download the App, and Get a Free Bumper Sticker! Get a hash tag. Develop a Facebook page and ask your friends to Like your vulnerability. (I might be exaggerating, but not by much.) Now, sit back and wait for the uninformed public to regurgitate the headlines about “New Vulnerability SUCKS!” If you are a security researcher who dreamed up all the above, start planning your speaking engagements on how the world as we know it will end, because (wait for it), “Everything SUCKS.”

Now is where the astute reader is thinking, “but wait a minute, isn’t it really a good thing to publicize the need to fix a widely-embedded library that is vulnerable?” Generally speaking, yes. Unfortunately, most of the publicity around some of these security vulnerabilities is out of proportion to the actual criticality and exploitability of the issues, which leads to customer panic. Customer panic is a good thing – sorta – if the vulnerability is the equivalent of the RMS Titanic’s “vulnerability” as exploited by a malicious iceberg. It’s not a good thing if we are talking about a rowboat with a bad case of chipped paint. The panic leads to suboptimal resource allocation as code providers (vendors and open source communities) are – to a point – forced to respond to these issues based on the amount of press they are generating instead of how serious they really are. It also means there is other more valuable work that goes undone. (Wouldn’t most customers actually prefer that vendors fix security issues in severity order instead of based on “what’s trending?”). Lastly, it creates a shellshock effect with customers, who cannot effectively deal with a continuous string of exaggerated vulnerabilities that cause their management to apply patches as soon as possible or document that their environment is free of the bug.

The relevant metric around how fast you fix things should be objective threat. If something has a Common Vulnerability Scoring System (CVSS) Base Score of 10, then I am all for widely publicizing the issue (with, of course, the Common Vulnerability Enumeration (CVE) number, so people can read an actual description, rather than “run for your lives, Godzilla is stomping your code!”) If something is CVSS 2, I really don’t care that it has a cuter critter than Bambi as a mascot and generally customers shouldn’t, either. To summarize my concerns, the willful marketing of security vulnerabilities is worrisome for security professionals because:

It creates excessive focus on issues that are not necessarily
truly critical

It creates grounds for confusion (as opposed to using CVEs)

It creates a significant support burden for organizations,* where resources would be better spent elsewhere

I would therefore, in the interests of constructive suggestions, recommend that customers assess the following criteria before calling all hands on deck over the next “branded” security vulnerability being marketed as the End of Life On Earth As We Know It:

1. Consider the source of the vulnerability information. There are some very good sites (arstechnica comes to mind) that have well-explained, readily understandable analyses of security issues. Obviously, the National Vulnerability Database (NVD) is also a great source of information.

2. Consider the actual severity of the bug (CVSS Base Score) and the exploitation scenario to determine “how bad is bad.”

3. Consider where the vulnerability exists, its implications, and whether mitigation controls exist in the environment: e.g., Heartbleed was CVSS 5.0, but the affected component (SSL), the nature of the information leakage (possible compromise of keys), and the lack of mitigation controls made it critical.

* e.g., businesses patching based on the level of hysteria rather than the level of threat

Organizations should look beyond cutesie vulnerability names so as to focus their attention where it matters most.  Inquiring about the most recent medium-severity bugs will do less in term of helping an organization secure its environment than, say applying existing patches for higher severity issues. Furthermore, it fosters a culture of “security by documentation” where organizations seek to collect information about a given bug from their cloud and software providers, while failing to apply existing patches in their environment. Nobody is perfect, but if you are going to worry, worry about vulnerabilities based on How Bad Is Bad, and not based on which ones have catchy acronyms, mascots or have generated a lot of press coverage.

Monday Mar 23, 2015

The Four Ps of Standards/Procurement Requirements/”Whatevahs”

I am a veteran – not merely a military veteran, but an information security veteran. I don’t get medals for the latter, but I do have battle scars. Many of the scars are relatively recent: a result of tearing my hair out from many, many, many mind-numbing reviews of publications, draft standards and other kinds of documents which are ostensibly meant to make security better, cybersecurity being “hot” and all. Alas, many of these documents have linguistic and operational difficulties that often make it highly unlikely that they will achieve their stated “better security” objectives.

After reviewing so many documents and running into common patterns, I decided to take a cue from my MBA days and categorize my concerns in a catchy way. Though not a marketing major, I vaguely recall the “four Ps” of marketing (product, price, place and promotion) and decided to adapt them to the world of standards/procurement requirements/whatevahs (which I will now refer to as SPW). They are:

Pr    Problem Statement
Precise Language and Scope
Pragmatic Solutions
Prescriptive Minimizations

I t     I offer the "four Ps of SPW" for those who are attempting to improve cybersecurity by fiat, or in other ways intended to compel the market, in hopes that we may collectively get to better security without sinking into the swamp of despair, dallying in the desert of dashed hopes, trekking through the tundra of too-obscure requirements (nice use of alliteration, no?) … you get the point. While I think my advice is generally applicable in the SPW (say “spew”) realm, the context for my discussion is assurance slash supply chain risk mitigation since that’s what I seem to review most often.

Problem Statement

I cannot tell you how many SPW documents I have read in which Someone Was Attempting to Make Someone Else Do Something More Securely, only it wasn’t clear what, exactly, or more importantly, why (or even that the requirements would result in “better security”). Anything that seeks to impose Something Security-Oriented On Someone needs a clear problem statement. Without this, a proposed SPW becomes an expensive wish list with no associated benefits to it. Ultimately, the seller has no idea what the buyer really wants or needs. If a government agency cannot explain what they are really worried about, in language the “comply-ee” can understand, they shouldn’t be surprised if they get a chocolate-covered cockroach (eew) when they ask for something sweet, crunchy and locally sourced. (I’d add “sustainable,” as there seems to be no shortages of cockroaches.)

With regard to security, “supply chain” has become the mantra for attempting to regulate almost 100% of what businesses do. Poor quality, “backdoor boogiemen,” assurance, “supply chain shutdown” are all very (very!) different problems. Worse, the ambiguity around proposing a standard for “supply chain security” may encompass 100% of business operations. Example: my employer does not make their own paper clips or wood stirrers for coffee cups. Do we really need to worry about a shortage of either? No? Then don’t describe “supply chain requirements” that ask technology suppliers to track the wood sourced for our coffee stirrers. Buying a poor quality product, for example, is a business risk. It’s not, per se, a supply chain risk. Furthermore, while poor quality may lead to poor security, not all security problems are a result of quality issues. Some are a result of buyers not understanding that commercial off-the-shelf (COTS) software, while general purpose and often very good, is not “all purpose” and not designed for all threat environments.

The second aspect of a problem statement is the provision of use cases. A use cases is a fancy way of saying, “for example.” Use cases are very important to help turn a problem statement into an “aha” moment for the reader. Moreover, use cases are important to limit scope and ensure that the SPW requirements are appropriate to serve its stated objectives. Absent a use case, you never really know what’s being asked for (and where it applies and where it does not apply). Use cases absolutely need to be contained within a requirements document.

For example, consider the US National Institute of Standards and Technology (NIST) Special Publication 800-152 A Profile for U.S. Federal Cryptographic Key Management Systems Draft 3 (December 2014). This special pub describes a combination of technical standards and policies around cryptographic key management systems. The problem is, nowhere in reading the document is it evident what, exactly, this applies to. Is this just “special, super secret key management systems for classified US government systems?” Or, does it apply to key management for things like Transport Layer Security (TLS) (or other cryptographic protocols that are well-established standards)? Why it matters: because if there are not use cases that define applicability, someone will assume it applies to everything. And, applying these requirements may conflict with (if not break) other standards.

90% of life isn’t showing up, it’s solving the right problem. You can’t solve the right problem if you don’t know (or cannot articulate) what it is, with some “for instances.”

Precise Language and Scope

It is astonishing to me how many SPW documents do not define core terminology used therein. Without a precise set of definitions, nobody really knows what is meant, and if something is vague, it’s going to be misinterpreted. (Worse, an undefined term may end up meaning whatever a “certifier” or other compliance overlord thinks it means: nobody ever really knows if they are compliant if compliant depends on what the certifier thinks it means.) Core terminology must be precisely and narrowly defined within the document. As the famous line goes from Let’s Call The Whole Thing Off,

“You like potato and I like potahto
You like tomato and I like tomahto
Potato, potahto, tomato, tomahto
Let’s call the whole thing off.” (Lyrics by Ira Gershwin, melody by George

The problem is, if a SPW is enshrined and applied, you can’t call it off. At least until the next revision. Figure out what to call a spud and make it clear, please!

For example, in the context of software, what is a vulnerability? A configuration error (leading to a security weakness)? A defect in software (that leads to a security weakness)? Any defect in software (regardless of the impact)? What if the design was intentional? Is a policy violation a vulnerability? A vulnerability cannot, surely, be all the above! And in fact, it isn’t, but just saying “vulnerability” and conflating all the above means that nobody will be able to come up with a remedy that works for all cases. (Note: for configurable software, if you configure it so my grandmother can hack into it, it’s not a “vulnerability,” it’s “user error.” There is only so much you can do to prevent a user shooting self in the foot when we are talking about firearms that allow you to point them at your feet.) Another example, what is a “module?” The answer may be very different depending on whether you are a hardware person or a software person.

If ‘it’ is not clear, ‘it’ is going to be misinterpreted.

Pragmatic Solutions

One of my biggest concerns with a lot of SPW documents is that they almost never take into account the value of pragmatism over perfection. Perfection is not achievable (much less at an acceptable cost) while “better” usually is achievable. (Surely “better” that everyone can do is better than “perfect” that is unachievable?) To those who insist, “evil slug vendors are profit driven and always want to do the minimum,” my response is that economics rules the world and doesn’t necessarily argue for the minimum. Generally speaking, it’s more profitable to find security vulnerabilities and fix them earlier in a product release cycle than waiting until you ship six affected versions of product and now have to produce 120 patches for a single issue (or patch 120 cloud instances). Most vendors know this (or find out the hard way). Customers certainly know this and complain if they have to apply too many patches (or if their cloud service uptime is negatively impacted by a lot of patch-related downtime).

More to the point, unless you can print money, invent a time machine or perfect cloning, time, money and people are always constrained resources so using them well is a must. Doing more X means – often – doing less of Y, because you can’t add more resource you don’t have or can’t find. Worse, doing more of X required for compliance may mean doing less of the Y that actually improves security, since they are mutually exclusive as long as resources are constrained and regulations are written by (or interpreted by) the Knights Who Say Ni.

In particular, I see little evidence that people proposing SPW have done much or any economic analysis of the cost of compliance. I know the government knows how to do this kind of analysis because – for example – the US Department of Defense does resource planning that among other things looks at “how many conflicts are we prepared to fight simultaneously?” rather than, “in a perfect world with unlimited resources and cyborg soldiers, we could take on Frabistatians, the Foobarians, and open a third front combating the Little Green Men from Marsians.” How I wish that other entities – any other entity – would analyze (e.g., do a reality check) on what the impact of X is before it becomes part of a SPW.

Any SPW should include an economic analysis of impact – and look at options. Included in that analysis should be the bane of (quasi-)regulatory ambition, “unintended consequences.” There are almost always unintended consequences of SPW, even those created with good motives. One of the big ones is, if you make it too expensive for suppliers to deal with you, there will be fewer suppliers. And that means choice will decrease and cost will increase. Any SPW should explicitly ask the question, “What would matter the most, be broadly implementable and cost the least (or be the most cost effective for all parties)?”

To provide an example, the NIST Interagency Report 7622 Notional Supply Chain Risk Management Practices for Federal Information Systems (the draft requirement has, I believe, since been excised) at one time wanted the “supplier” (e.g., a vendor) to notify the acquirer (e.g., a government agency) of “all personnel changes involving maintenance.” I suspect that the intent was something to the effect that, if the acquirer (let’s say, DoD) outsources a service, and that service involves a fundamental change of venue – e.g., the maintenance for the US Department of Defense manpower system is outsourced to Hostile Foreign Country, DoD wants to be notified. However, that is not what the requirement stated. One interpretation would be that any time someone touched code who didn’t write the original code (“a personnel change involving maintenance”) that a vendor would have to notify the government. Ok, Oracle has almost 5000 products (and lots and lots of clouds), billions of lines of code, and every day there are a lot of code checkouts where someone is changing something he or she did not write. Are we supposed to tweet all that stuff? What is that going to do for the acquirer? “Kaitlyn checked out and changed code that, like, Ashley wrote, LOL, OMG!”

Figure out what you really want, and what it is worth to you to get it.

Prescriptive Minimization

With rare exceptions, non-technical* process or management standards should not tell industry how exactly to do something, if for no other reason than there is no such thing as “best practice.” There are certainly better or worse practices, but arguably no single practice that everyone does, exactly the same way, that will work equally well for everyone subject to the requirements, for any length of time. Worse, SPW diktats often stifle innovation, drive up costs (without commensurate benefit) and fall prey to the buggy whip effect (where you are specifying how to use buggy whips long after people have moved from horse-and-buggy to Model Ts - or better). Add to all these reasons the economic impact referenced above.

To provide one example, consider (draft) NIST Special Publication 800-160 Systems Security Engineering, containing a requirement that, in the event of a discovered security bug, the engineering team should conduct root cause analysis. This sounds like a Mom and Apple Pie requirement on the face of it, so what could possibly be wrong with that? A clear Best Practice, right? Well, no, not really, on grounds of pragmatism and context.

Consider a security bug that is not only high impact but for which there is an exploit circulating in the wild. For commercial software vendors, job 1 will be getting a patch into customers’ hands (or at least the hands of their customers’ system administrators) and/or patching their cloud instances, as the case may be. Protection of customers under these circumstances is initially way more important than determining causation.

Second, it doesn’t necessarily make sense to do a root cause analysis on every single security bug of every severity. What does make sense is to deep dive on the more severe bugs (e.g., high Common Vulnerability Scoring System (CVSS) Base Score bugs), because those are the ones you really want to ensure you fixed completely (and avoid in the future). You might want to ask the following as part of your analysis:

“How/when did this get into the code base?”
“What is the resulting vulnerability (how can it be exploited)?”
“Have we looked elsewhere for similar problems?”
“Have we added test cases to regression tests and other test suites (like static analysis tools) to ensure that we can automate finding other instances?”
“Have we fixed it everywhere (or everywhere that is relevant?)”and
“Have we attempted to enshrine/transfer knowledge of the severity and impact of this bug across the development organization (so everyone knows why it’s a big deal and how to avoid it in future)?”

Given scarce resources, I’d argue that root cause analysis on a CVSS 0 bug is not as important as thoroughly addressing – and in future avoiding – a CVSS 9.0 or 10.0 bug, along the lines of the above analysis. If a standard enshrines the former, it leads to suboptimal resource allocation (like spreading peanut butter over too many slices of bread). Worse, any company doing the “better” thing will get dinged as being non-standards compliant if there is a Best Practice enshrined in SPW that calls for root cause analysis of everything, regardless of severity. Perfection works against actual security improvement.

Another “best practice” I see shilled relentlessly is third party static analysis. I’ve opined on why that is not a best practice in previous blogs, but I have new reasons to avoid it like the plague it is, which is a real world example of the high cost and low utility. Recently, we were made aware that a customer of Oracle (without asking our permission, that we would not have given if asked) submitted our software to a third party that does static analysis on binaries. Where to start with how extremely bad this is? Numero uno: the customer violated their license agreement with Oracle, which alone made their actions completely unacceptable. Add to that, the report we were furnished included alleged vulnerabilities not merely in Oracle but in another product Not Made By Oracle. (Needless to say, we could neither analyze those issues nor fix them in the event they turned out to be actual vulnerabilities and really, we did not want to see alleged vulnerabilities in Someone Else’s Code. That information is extremely sensitive and should not have been given to us.) Last but far from least was the fact that – drum roll – not one of the alleged security issues the third party reported was, in fact, an actual security vulnerability. 0% accuracy: zilch, zip, nada, bubkes, a’ohe mea. Further, one of our best security leads (I’d bill him out at least $2,000 bucks an hour) wasted his very valuable time determining that there was “no there, there.”

Running a tool (if and only if you have permission to do it) is nothing; the ability to analyze the results is everything. Third parties cannot do that since they have no actual code knowledge of what they are running the tool on, especially not on a code base as big as Oracle’s is. Third party static analysis is thus only a best practice if you want to waste time and money. But it’s the vendor’s time that is being wasted (maybe that third party should reimburse us the $2K an hour our kahuna spent analyzing their errata?), and the customer’s money. And last, but really first, violating licensing terms is unacceptable business conduct.


Nobody is perfect, but with all the attention being focused on cybersecurity, it would be really helpful if attempted problem solvers writing SPW could sharpen their – I was going to say, knives, but I am not sure I mean that! – focus. Yes, a sharpened focus is what is needed. Cybersecurity is an important area. Better security is achievable, but only if we know what we are worried about, we speak the same language, we can look at relative costs and benefits, and we allow for latitude in how we get to better. We can’t do everything, but everybody can do something. Let’s do the some of the things that matter – and that won’t make us spend resources checking boxes instead of making sure nobody can break into the boxes.

· I    * I note that one reason for technical standards is, of course, interoperability. In which case, people do need to implement, say, the Secure Whateverworks Protocol (SWP) a particular way, or it won’t work with another vendor’s implementation of SWP.

For More Information

Ruthlessly self-serving announcement follows: my sister and I, writing as Maddi Davidson, are pleased to announce that we have completed our third book in the Miss-Information Technology Mystery Series, With Murder You Get Sushi. (Also, our short story “Heartfelt” will appear in Mystery Times Ten this month, published by Buddhapuss Ink.)

Apropos of nothing having to do with security, I have discovered and become totally addicted to The Palliser Novels by Anthony Trollope. Like high class soap opera, only you get classics points for reading them. (Best of all, nobody in the book is named “Kardashian.”)

Tuesday Mar 11, 2014

Mandated Third Party Static Analysis: Bad Public Policy, Bad Security

Many commercial off-the-shelf (COTS) vendors have recently seen an uptick of interest by their customers in third party static analysis or static analysis of binaries (compiled code). Customers who are insisting upon this in some cases have referenced the (then-)SANS Top Twenty Critical Controls ( to support their position, specifically, Critical Control 6, Application Software Security:

"Configuration/Hygiene: Test in-house developed and third-party-procured web and other application software for coding errors and malware insertion, including backdoors, prior to deployment using automated static code analysis software. If source code is not available, these organizations should test compiled code using static binary analysis tools (emphases added). In particular, input validation and output encoding routines of application software should be carefully reviewed and tested."

Recently, the "ownership" of the 20 Critical Controls has passed to the Council on CyberSecurity and the particular provision on third party static analysis has been excised. Oracle provided feedback on the provision (more on which below) and appreciates the responsiveness of the editors of the 20 Critical Controls to our comments.

The argument for third party code analysis is that customers would like to know that they are getting “reasonably defect-free” code in a product. All things being equal, knowing you are getting better quality code than not is a good thing, while noting that there is no defect-free or even security defect-free software – no tool finds all problems and they generally don’t find design defects. Also, a product that is “testably free” of obvious defects may still have significant security flaws – like not having any authentication, access control, or auditing. Nobody is arguing that static analysis isn’t a good thing – that’s why a lot of vendors already do it and don’t need a third party to “attest” to their code (assuming there is a basis for trusting the third party other than their saying “trust us”).

Oracle believes third party static analysis is at best infeasible for organizations with mature security assurance practices and – well, a bad idea, not to put too fine a point on it. The reasons why it is a bad idea are expanded upon in detail below, and include: 1) worse, not better security 2) increased security risk to customers, 3) an increased risk of intellectual property theft and 4) increased costs for commercial software providers without a commensurate increase in security. Note that this discussion does not address the use of other tools - such as so-called web vulnerability analysis tools – that operate against “as installed” object code. These tools also have challenges (i.e., a high rate of false positives) but do not in general pose the same security threats, risks and high costs that static analysis as conducted by third parties does.

Discussion: Static analysis tools are one of many means by which vendors of software, including commercial off-the-shelf (COTS) software, can find coding defects that may lead to exploitable security vulnerabilities. Many vendors – especially large COTS providers - do static analysis of their own code as part of a robust, secure software development process. In fact, there are many different types of testing that can be done to improve security and reliability of code, to include regression testing (ensuring that changes to code do not break something else, and that code operates correctly after it has been modified), “fuzzing” tools, web application vulnerability tools and more. No one tool finds all issues or is necessarily even suitable for all technologies or all programming languages. Most companies use a multiplicity of tools that they select based on factors such as cost, ease-of-use, what the tools find, how well and how accurately, programming languages the tool understands, and other factors. Note of course that these tools must be used in a greater security assurance context (security training, ethical hacking, threat modeling, etc.), echoing the popular nostrum that security has to be “baked in, not bolted on.” Static analysis and other tools can’t “bake in” security – just find coding errors that may lead to security weaknesses. More to the point, static analysis tools should correctly be categorized as “code analysis tools” rather than “code testing tools,” because they do not automatically produce accurate or actionable results when run and cannot be used, typically, by a junior developer or quality assurance (QA) person.

These tools must in general be “tuned” or “trained” or in some cases “programmed” to work against a particular code base, and thus the people using them need to be skilled developers, security analysts or QA experts. Oracle has spent many person years evaluating the tools we use, and have made a significant commitment to a particular static analysis tool which works the best against much – but not all – of our code base. We have found that results are not typically repeatable from code base to code base even within a company. That is, just because the tool works well on one code base does not mean it will work equally well on another product -- another reason to work with a strong vendor who will consider improving the tool to address weaknesses. In short, static analysis tools are not a magic bullet for all security ills, and the odds of a third party being able to do meaningful, accurate and cost-effective static code analysis are slim to none.

1. Third party static analysis is not industry-standard practice.
Despite the marketing claims of the third parties that do this, “third party code review” is not “industry best practice.” As it happens, it is certainly not industry-standard practice for multiple reasons, not the least of which is the lack of validation of the entities and tools used to do such “validation” and the lack of standards to measure efficacy, such as what does the tool find, how well, and how cost effectively? As Juvenal so famously remarked, “Quis custodiet ipsos custodes?” (Who watches the watchmen?) Any third party can claim, and many do, that “we have zero false positives” but there is no way to validate such puffery – and it is puffery. (Sarcasm on: I think any company that does static analysis as a service should agree to have their code analyzed by a competitor. After all, we only have Company X’s say-so that they can find all defects known to mankind with zero false positives, whiten your teeth and get rid of ring-around-the-collar, all with a morning-fresh scent!)

The current International Standards Organization (ISO) standard for assurance (which encompasses the validation of secure code development), the international Common Criteria (ISO-15408), is, in fact, retreating from the need for source code access currently required at higher assurance levels (e.g., Evaluation Assurance Level (EAL) 4). While limited vulnerability analysis has been part of higher assurance evaluations currently being deprecated by the U.S. National Information Assurance Partnership (NIAP), static analysis has not been a requirement at commercial assurance levels. Hence, “the current ISO assurance standard” does not include third party static code analysis and thus, “third party static analysis” is not standard industry practice. Lastly, “third party code analysis” is clearly not “industry best practice” if for no other reason than all the major COTS vendors are opposed to it and will not agree to it. We are already analyzing our own code, thanks very much.

(It should be noted that third party systematic manual code review is equally impractical for the code bases of most commercial software. The Oracle database, for example, has multiple millions of lines of code. Manual code review for the scale of code most COTS vendors produce would accomplish little except pad the bank accounts of the consultants doing it without commensurate value provided (or risk reduction) for either the vendor or the customers of the product. Lastly, the nature of commercial development is that the code is continuously in development: the code base literally changes daily. Third party manual code review in these circumstances would accomplish absolutely nothing. It would be like painting a house while it is under construction.)

2. Many vendors already use third party tools to find coding errors that may lead to exploitable security vulnerabilities.
As noted, many large COTS vendors have well-established assurance programs that include the use of a multiplicity of tools to attempt to find not merely defects in their code, but defects that lead to exploitable security vulnerabilities. Since only a vendor can actually fix a product defect in their proprietary code, and generally most vulnerabilities need a “code fix” to eliminate the vulnerability, it makes sense for vendors to run these tools themselves. Many do.

Oracle, for example, has a site license for a COTS static analysis tool and Oracle also produces a static analysis tool in-house (Parfait, which was originally developed by Sun Labs). With Parfait, Oracle has the luxury of enhancing the tool quickly to meet Oracle-specific needs. Oracle has also licensed a web application vulnerability testing tool, and has produced a number of in-house tools that focus on Oracle’s own (proprietary) technologies. It is unlikely that any third party tool can fuzz Oracle PL/SQL as well as Oracle’s own tools, or analyze Oracle’s proprietary SQL networking protocol as well as Oracle’s in-house tools do. The Oracle Ethical Hacking Team (EHT) also develops tools that they use to “hack” Oracle products, some of which are “productized” for use by other development and QA teams. As Oracle runs Oracle Corporation on Oracle products, Oracle has a built-in incentive to write and deliver secure code. (In fact, this is not unusual: many COTS vendors run their own businesses on their own products and are thus highly motivated to build secure products. Third party code testers typically do not build anything that they run their own enterprises on.)

The above tool usage within Oracle is in addition to extensive regression testing of functionality to high levels of code coverage, including regression testing of security functionality. Oracle also uses other third party security tools (many of which are open source) that are vetted and recommended by the Oracle Software Security Assurance (OSSA) team. Additionally, Oracle measures compliance with “use of automated tools” as part of the OSSA program. Compliance against OSSA is reported quarterly to development line-of-business owners as well as executive management (the company president and the CEO). Many vendors have similarly robust assurance programs that include static analysis as one of many means to improve product security.

Several large software vendors have acquired static analysis (or other) code analysis tools. HP, for example, acquired both Fortify and WebInspect and IBM acquired Coverity. This is indicative both of these vendors’ commitment to “the secure code marketplace” but also, one assumes, to secure development within their own organizations. Note that while both vendors have service offerings for the tools, neither is pushing “third party code testing,” which says a lot. Everything, actually.

Note that most vendors will not provide static analysis results to customers for valid business reasons, including ensuring the security of all customers. For example, a vendor who finds a vulnerability may often fix the issue in the version of product that is under development (i.e., the “next product train leaving the station”). Newer versions are more secure (and less costly to maintain since the issue is already fixed and no patch is required). However, most vendors do not - or cannot - fix an issue in all shipping versions of product and certainly not in versions that have been deprecated. Telling customers the specifics of a vulnerability (i.e., by showing them scan results) would put all customers on older, unfixed or deprecated versions at risk.

3. Testing COTS for coding errors and malware insertion, including backdoors, prior to deployment using automated static code analysis software increases costs without a commensurate return on investment (ROI).
The use of static code analysis software is a highly technical endeavor requiring skilled development personnel. There are skill requirements and a necessity for detailed operational knowledge of how the software is built to help eliminate false positives, factors that raise the cost of this form of “testing.” Additionally, static code analysis tools are not the tool of choice for detecting malware or backdoors. (It is in fact, trivial, to come up with a “backdoor” that, if inserted into code, would not be detected by even the best static analysis tools. There was an experiment at Sandia Labs in which a backdoor was inserted into code and code reviewers told where in code to look for it. They could not find it – even knowing where to look.)

If the real concern of a customer insisting on a third party code scan is malware and backdoor detection: it won’t work and thus represents an extremely expensive – and useless – distraction.

4. Third party code analysis will diminish overall product security.
It is precisely leading vendors’ experience with static analysis tools that contributes to their unwillingness to have third parties attempt to analyze code – emphasis on “attempt.” None of these tools are “plug and play": in some cases, it has taken years, not months, to be able to achieve actionable results, even from the best available static analysis tools. These are in fact code analysis tools and must be “tuned” – and in some cases actually “programmed”- to understand code, and must typically be run by an experienced developer (that is, a developer who understand the particular code base being analyzed) for results to be useful and actionable. There are many reasons why static analysis tools either raise many false positives, or skip entire bodies of code. For example, because of the way Oracle implements particular functionality (memory management) in the database, static analysis tools that look for buffer overflows either do not work, or raise false positives (Oracle writes its own checks to look for those issues).

The rate of false positives from use of a “random” tool run by inexperienced operators – especially on a code base as large as that of most commercial products – would put a vendor in the position of responding to unsubstantiated fear, uncertainty, and doubt (FUD). In a code base of 10,000,000 lines of code, even a false positive rate of one per 1000 lines of code would yield 10,000 “false positives” to chase down. The cost of doing this is prohibitive. (One such tool run against a large Oracle code base generated a false positive for every 3.4 lines of code, or about 160,000 false positives in toto due to the size of the code base.)

This is why most people using these tools must “tune” them to drown out “noise.” Many vendors have already had this false positive issue with customers running web application vulnerability tools and delivering in some cases hundreds of pages of “alarms” in which there were, perhaps, a half page of actionable issues. The rate of false positives is the single biggest determinant whether these tools are worth using or an expensive distraction (aka “rathole”).

No third party firm has to prove that their tool is accurate – especially not if the vendor is forced to use a third party to validate their code – and thus there is little to no incentive to improve their tool. Consultants get paid more the longer they are on site and working. A legislative or “standards” requirement for “third party code analysis” is therefore a license for the third party doing it to print money. Putting it differently, if the use of third party static analysis was accurate and cost effective, why wouldn’t vendors already be doing it? Instead, many vendors use static analysis tools in-house, because they own the code, and are willing to assume the cost of going up the learning curve for a long term benefit to them of reduced defects (and reduced cost of fixing these defects as more vulnerabilities are found earlier in the development cycle).

In short, the use of a third party is the most expensive, non-useful, high-cost attempt at “better code” most vendors could possibly use, and would result in worse security, not better security as in-house “security boots on the ground” are diverted to working with the third party. It is unreasonable to expect any vendor to in effect tune a third party tool and train the third party on their code – and then have to pay the third party for the privilege of doing it. Third party static analysis represents an unacceptably high opportunity cost caused by the “crowding out effect” of taking scarce security resources and using them on activity of low value to the vendor and to their customers. The only “winner” here is the third party. Ka-chink. Ka-chink.

5. Third party code analysis puts customers at increased risk.
As noted, there is no standard for what third party static analysis tools find, let alone how well and how economically they find it. More problematically, there are no standards for protection of any actual vulnerabilities these tools find. In effect, third party code analysis allows the third party to amass a database of unfixed vulnerabilities in products without any requirements for data protection or any recourse should that information be sold, incorporated into a hacking tool or breached. The mere fact of a third party amassing such sensitive information makes the third party a hacker target. Why attempt to hack products one by one if you can break into a third party’s network and get a listing of defects across multiple products – in the handy “economy size?” The problem is magnified if the “decompiled” source code is stored at the third party: such source code would be an even larger hacker target than the list of vulnerabilities the third party found.

Most vendors have very strict controls not merely on their source code, but on the information about product vulnerabilities that they know about and are triaging and fixing. Oracle Corporation, for example, has stringent security vulnerability handling policies that are promulgated and “scored” as part of Oracle’s software and hardware assurance program. Oracle uses its own secure database technology (row level access control) to enforce “need to know” on security vulnerabilities, information that is considered among the most sensitive information the company has. Security bugs are not published (meaning, they are not generally searchable and readable across the company or accessible by customers). Also, security bug access is stringently limited to those working on a bug fix (and selected others, such as security analysts and the security point of contact (SPOC) for the development area).

One of the reasons Oracle is stringent about limiting access to security vulnerability information is that this information often does leak when “managed” by third parties, even third parties with presumed expertise in secret-keeping. In the past, MI5 circulated information about a non-public Oracle database vulnerability among UK defense and intelligence entities (it should be noted that nobody reported this to Oracle, despite the fact that only Oracle could issue a patch for the issue). Oracle was only notified about the bug by a US commercial company to whom the information had leaked. As the saying goes, two people can’t keep a secret.

There is another risk that has not generally been considered in third party static analysis, and that is the increased interest in cyber-offense. There is evidence that the market for so-called zero-day vulnerabilities is being fueled in part by governments seeking to develop cyber-offense tools. (StuxNet, for example, allegedly made use of at least four “zero-day” vulnerabilities: that is, vulnerabilities not previously reported to a vendor.) Coupled with the increased interest in military suppliers/system integrators in getting into the “cyber security business,” it is not a stretch to think that at last some third parties getting into the “code analysis” business can and would use that as an opportunity to “sell to both sides” – use legislative fiat or customer pressure to force vendors to consent to static analysis, and then surreptitiously sell the vulnerabilities they found to the highest bidder as zero-days. Who would know?

Governments in particular cannot reasonably simultaneously fuel the market in zero days, complain at how irresponsible their COTS vendors are for not building better code and/or insist on third party static analysis. This is like stoking the fire and then complaining that the room is too hot.

6. Equality of access to vulnerability information protects all customers.
Most vendors do not provide advance information on security vulnerabilities to some customers but not others, or more information about security vulnerabilities to some customers but not others. As noted above, one reason for this is the heightened risk that such information will leak, and put the customers “not in the know” at increased risk. Not to mention, all customers believe their secrets are as worthy of protection as any other customer: nobody wants to be on the “Last Notified” list.

Thus, third party static analysis is problematic because it may result in violating basic fairness and equality in terms of vulnerability disclosure in the rare instances where these scans actually find exploitable vulnerabilities. The business model for some vendors offering static analysis as a service is to convince the customers of the vendor that the vendor is an evil slug and cannot be trusted, and thus the customer should insist on the third party analyzing the vendors’ code base.

There is an implicit assumption that the vendor will fix vulnerabilities that the third party static analysis finds immediately, or at least, before the customer buys/installs the product. However, the reality is more subtle than that. For one thing, it is almost never the case that a vulnerability exists in one and only one version of product: it may also exist on older versions. Complicating the matter: some issues cannot be “fixed” via a patch to the software but require the vendor to rearchitect functionality. This typically can only be done in so-called major product releases, which may only occur every two to three years. Furthermore, such issues often cannot be fixed on older versions because the scope of change is so drastic it will break dependent applications. Thus, a customer (as well as the third party) has information about a “not-easily-fixed” vulnerability which puts other customers at a disadvantage and at risk to the extent that information may leak.

Therefore, allowing some customers access to the results of a third party code scan in advance of a product release would violate most vendors’ disclosure policies as well as actually increasing risk to many, many customers, and potentially that increased risk could exist for a long period of time.

7. Third party code analysis sets an unacceptable precedent that risks vendors’ core intellectual property (IP).
COTS vendors maintain very tight control over their proprietary source code because it is core, high-value IP. As such, most COTS vendors will not allow third parties to conduct static analysis against source code (and for purposes of this discussion, this includes static analysis against binaries, which typically violates standard license agreements).

Virtually all companies are aware of the tremendous cost of intellectual property theft: billions of dollars per year, according to published reports. Many nation states, including those that condone if not encourage wholesale intellectual property theft, are now asking for source code access as a condition of selling COTS products into their markets. Most COTS vendors have refused these requests. One can easily imagine that for some nation states, the primary reason to request source code access (or, alternatively, “third party analysis of code”) is for intellectual property theft or economic espionage. Once a government-sanctioned third party has access to source code, so may the government. (Why steal source code if you can get a vendor to gift wrap it and hand it to you under the rubric of “third party code analysis?”)

Another likely reason some governments may insist on source code access (or third party code analysis) is to analyze the code for weaknesses they then exploit for their own national security purposes (e.g., more intellectual property theft). All things being equal, it is easier to find defects in source code than in object code. Refusing to accede to these requests – in addition to, of course, a vendor doing its own code analysis and defect remediation – thus protects all customers. In short, agreeing to any third party code analysis involving source code – either static analysis or static analysis of binaries - would make it very difficult if not impossible for a vendor to refuse any other similar requests for source code access, which would put their core intellectual property at risk. Third party code analysis is a very bad idea because there is no way to “undo” a precedent once it is set.

Software should have a wide variety of tests performed before it is shipped and additional security tests (such as penetration tests) should be used against “as-deployed” software. However, the level of testing should be commensurate with the risk, which is both basic risk management and appropriate (scarce) resource management. A typical firm has many software elements, most probably COTS, and to suggest that they all be tested with static analysis tools begs a sanity check. The scope of COTS alone argues against this requirement: COTS products run the gamut from operating systems to databases to middleware, business intelligence and other analytic tools, business applications (accounting, supply chain management, manufacturing) as well as specialized vertical market applications (e.g., clinical trial software),
representing a number of programming languages and billions – no, hundreds of billions - of lines of code.

The use of static analysis tools in development to help find and remediate security vulnerabilities is a good assurance practice, albeit a difficult one because of the complexity of software and the difficulty of using these tools. The only utility of these tools is that they be used by the producer of software in a cost- effective way geared towards sustained vulnerability reduction over time. The mandated use of third party static analysis to “validate” or “test” code is unsupportable, for reasons of cost (especially opportunity cost), precedence, increased risk to vendors’ IP and increased security risk to customers. The third party static code analysis market is little more than a subterfuge for enabling the zero-day vulnerability market: bad security, at a high cost, and very bad public policy.

Book of the Month
It’s been so long since I blogged, it’s hard to pick out just a few books to recommend. Here are three, and a "freebie":

Hawaiki Rising: Hōkūle’a, Nainoa Thompson and the Hawaiian Renaissance by Sam Low
Among the most amazing tidbits of history are the vast voyages that the Polynesians made to settle (and travel among) Tahiti, Hawai’i and Aotearoa (New Zealand) using navigational methods largely lost to history. (Magellan – meh – he had a compass and sextant.) This book describes the re-creation of Polynesian wayfinding in Hawai’i in the 1970s via the building of a double-hulled Polynesian voyaging canoe, the Hōkūle’a, and how one amazing Hawaiian (Nainoa Thompson) – under the tutelage of one of the last practitioners of wayfinding (Mau Piailug) – made an amazing voyage from Hawai’i to Tahiti using only his knowledge of the stars, the winds, and the currents. (Aside: one of my favorite songs is “H
ōkūle’a Hula,” which describes this voyage, and is so nicely performed by Erik Lee.) Note: the Hōkūle’a is currently on a voyage around the world.

The Korean War by Max Hastings
Max Hastings is one of the few historians whom I think is truly balanced: he looks at the moral issues of history, weighs them, and presents a fair analysis – not “shove-it-down-your-throat revisionism.” He also makes use of a lot of first-person accounts, which makes history come alive. The Korean War is in so many cases a forgotten war, especially the fact that it literally is a war that never ended. It’s a good lesson of history, as it is made clear that the US drew down their military so rapidly and drastically after the World War II that we were largely (I am trying not to say “completely”) unprepared for Korea. (Moral: there is always another war.)

Code Talker by Chester Nez
Many people now know of the crucial role that members of the Navajo Nation played in the Pacific War: the code they created that provided a crucial advantage (and was never broken). This book is a first-person account of the experiences of one Navajo code talker, from his experiences growing up on the reservation to his training as a Marine, and his experiences in the Pacific Theater. Fascinating.

Securing Oracle Database 12c: A Technical Primer
If you are a DBA or security professional looking for more information on Oracle database security, then you will be interested in this book. Written by members of Oracle's engineering team and the President of the International Oracle User Group (IOUG), Michelle Malcher, the book provides a primer on capabilities such as data redaction, privilege analysis and conditional auditing. If you have Oracle databases in your environment, you will want to add this book to your collection of professional information. Register now for the complimentary eBook and learn from the experts.

Monday May 13, 2013

I Love Standards…There Are So Many Of Them

The title is not an original bon mot by me – it’s been said often, by others, and by many with more experience than I have in developing standards.  It is with mixed emotions that I feel compelled to talk about a (generally good and certainly well-intentioned) standards organization: the US National Institute of Standards and Technology (NIST). I should state at the outset that I have a lot of respect for NIST. In the past, I have even urged a Congressional committee (House Science and Technology, if memory serves) to try to allocate more money to NIST for cybersecurity standards work.  I’ve also met a number of people who work at NIST – some of whom have since left NIST and brought their considerable talents to other government agencies, one of whom I ran into recently and mentioned how I still wore a black armband years after he had left NIST because he had done such great work there and I missed working with him. All that said, I’ve seen a few trends at NIST recently that are – of concern.

When in Doubt, Hire a Consultant

I’ve talked in other blog entries about the concern I have that so much of NIST’s outwardly-visible work seems to be done not by NIST but by consultants. I’m not down on consultants for all purposes, mind you – what is having your tires rotated and your oil changed except “using a car consultant?” However, in the area of developing standards or policy guidance it is of concern, especially when, as has been the case recently, the number of consultants working on a NIST publication or draft document is greater than the number of NIST employees contributing to it.  There are business reasons, often, to use consultants. But you cannot, should not, and must not “outsource” a core mission, or why are you doing it? This is true in spades for government agencies.  Otherwise, there is an entire beltway’s worth of people just aching to tell you about a problem you didn’t know you had, propose a standard (or regulation) for it, write the standard/regulation, interpret it and “certify” that Other Entities meet it. To use a song title, “Nice Work If You Can Get It.”* Some recent consultant-heavy efforts are all over the map, perhaps because there isn’t a NIST employee to say, "you say po-TAY-to, I sy po-TAH-to, let's call the whole thing off." ** (Or at least make sure the potato standard is Idaho russet – always a good choice.)

Another explanation – not intentionally sinister but definitely a possibility – is that consultants’ business models are often tied to repeat engagements.  A short, concise, narrowly-tailored and readily understandable standard isn’t going to generate as much business for them as a long, complex and “subject to interpretation – and five people will interpret this six different ways” – document. 

In short: I really don’t like reading a document like NISTIR 7622 (more on which below) where most of the people who developed it are consultants. NIST’s core mission is standards development: NIST needs to own their core mission and not farm it out.

Son of FISMA

I have no personal experience with the Federal Information Security Management Act of 2002 (FISMA) except the amount of complaining I hear about it second hand, which is considerable.  The gist of the complaints is that FISMA asks people to do a lot of stuff that looks earnestly security oriented, not all of which is equally important.

Why should we care? To quote myself (in an obnoxiously self-referential way): “time, money and (qualified security) people are always limited.” That is, the more security degenerates into a list of the 3000 things you Must Do To Appease the Audit Gods, the less real security we will have (really, who keeps track of 3000 Must Dos, much less does them? It sounds like a demented Girl Scout merit badge). And, in fact, the one thing you read about FISMA is that many government agencies aren’t actually compliant because they missed a bunch of FISMA checkboxes.  Especially since knowledgeable resources (that is, good security people) are limited, it’s much better to do the important things well then maintain the farce that you can check 3000 boxes, which certainly cannot all be equally important. (It’s not even clear how many of these requirements contribute to actual security as opposed to supporting the No Auditor Left Behind Act.)

If the scuttlebutt I hear is accurate, the only thing that could make FISMA worse is – you guessed it –adding more checkboxes. It is thus with considerable regret that I heard recently that NIST updated NIST Special Publication 800-53 (which NIST has produced as part of its statutory responsibilities under FISMA). The Revision 4 update included more requirements in the area of supply chain risk management and software assurance and trustworthiness.  Now why would I, a maven of assurance, object to this? Because a) we already have actual standards around assurance, b) having FISMA-specific requirements means that pretty much every piece of Commercial Off-the-Shelf (COTS) software will have to be designed and built to be FISMA compliant or COTS software/hardware vendors can’t sell into the Federal government and (c) we don’t want a race by other governments to come up with competing standards, to the point where we’re checking not 3000 but 9000 or 12000 boxes and probably can’t come up with a single piece of COTS globally let alone one that meets all 12000 requirements. (Another example is the set of supply chain/assurance requirements in the telecom sector in India that include a) asking for details about country of origin and b) specific contractual terms that buyers anywhere in the supply chain are expected to use.  An unintended result is that a vendor will need to (a) disclose sensitive supply chain data (which itself may be a trade secret) and (b) modify processes around global COTS to sell into one country.)

Some of the new NIST guidance is problematic for any COTS supplier. To provide one example, consider:

“The artifacts generated by these development activities (e.g., functional specifications, high-level/low-level designs, implementation representations [source code and hardware schematics], the results from static/dynamic testing and code analysis (emphasis mine)) can provide important evidence that the information systems (including the components that compose those systems) will be more reliable and trustworthy. Security evidence can also be generated from security testing conducted by independent, accredited, third-party assessment organizations (e.g., Common Criteria Testing Laboratories (emphasis mine), Cryptographic/Security Testing Laboratories, and other assessment activities by government and private sector organizations.)”

For a start, to the extent that components are COTS, such “static testing” is certainly not going to happen by a third party nor will the results be provided to a customer. Once you allow random customers – especially governments – access to your source code or to static analysis results, you might as well gift wrap your code and send it to a country that engages in industrial espionage, because no vendor, having agreed to this for one government, will ever be able to say no to Nation States That Steal Stuff.  (And static analysis results, to the extent some vulnerabilities are not fixed yet, just provide hackers a road map for how and where to break in.) Should vendors do static analysis themselves? Sure, and many do. It’s fair for customers to ask whether this is done, and how a supplier ensures that the worst stuff is fixed before the supplier ships product. But it is worth noting – again – that if these tools were easy to use and relatively error free, everyone would be at a high level of tools usage maturity years ago. Using static analysis tools is like learning Classic Greek – very hard, indeed. (OK, koinic Greek isn’t too bad but Homeric Greek or Linear B, fuhgeddabout it.)

With reference to the Common Criteria (CC), the difficulty now is that vendors have a much harder time doing CC evaluations than in the past because of other forces narrowing CC evaluations into a small set of products that have Protection Profiles (PPs). The result has been and will be for the foreseeable future – fewer evaluated products. The National Information Assurance Partnership (NIAP) – the US evaluation scheme – has ostensibly good reasons for their “narrowed/focused” CC-directions. But it is more than a little ironic that the NIST 800-53 revision should mention CC evaluations as an assurance measure at a time when the pipeline of evaluated products is shrinking, in large part due to the directions taken by another government entity (NIAP). What is industry to make of this apparent contradiction? Besides corporate head scratching, that is.

There are other – many other sections – I could comment upon, but one sticks out as worthy of notice:

“Supply chain risk is part of the advanced persistent threat (APT).”

It’s bad enough that “supply chain risk” is such a vague term that it encompasses basically any and all risk of buying from a third party. (Including “buying a crummy product” which is not actually a supply chain-specific risk but a risk of buying any and all products.) Can bad guys try to corrupt the supply chain? Sure. Does that make any and all supply chain risks “part of APT?” Heck, no. We have enough hysteria about supply chain risk and APT without linking them together for Super-Hysteria. 

To sum up, I don’t disagree that customers in some cases – and for some, not all applications – may wish higher levels of assurance or have a heightened awareness of cyber-specific supply chain threats (e.g., counterfeiting and deliberate insertion of malware in code). However, incorporation of supply chain provisions and assurance requirements into NIST 800-53 has the unintended effect of requiring any and all COTS products to be sold to government agencies – which is all of them as far as I know – to be subject to FISMA.

What if the state of Idaho decided that every piece of software had to attest to the fact that No Actual Moose were harmed during the production of this software and that any moose used in code production all had background checks? What if every other state enumerated specific assurance requirements and specific supply chain risk management practices? What if they conflict with each other, or with the NIST 800-53 requirements? I mean really, why are these specific requirements called out in NIST 800-53 at all? There really aren’t that many ways to build good software.  FISMA as interpreted by NIST 800-53 really, really shouldn’t roll its own.

IT Came from Outer Space – NISTIR 7622

I’ve already opined at length about how bad the NIST Interagency Report (NISTIR) 7622 is. I had 30 pages of comments on the first 80-page draft. The second draft only allowed comments of the Excel Spreadsheet form:  “Section A.b, change ‘must’ to ‘should,’ for the reason ‘because ‘must’ is impossible’” and so on. This format didn’t allow for wholesale comments such as “it’s unclear what problem this section is trying to solve and represents overreach, fuzzy definition and fuzzier thinking.”  NISTIR 7622 was and is so dreadful that an industry association signed a letter that said, in effect, NISTIR 7622 was not salvageable, couldn’t be edited to something that could work, and needed to be scrapped in toto.

I have used NISTIR 7622 multiple times as a negative example: most recently, to an audience of security practitioners as to why they need to be aware of what regulations are coming down the pike and speak up early and often.  I also used it in the context of a (humorous) paper I did at the recent RSA Conference with a colleague, the subject of which was described as “doubtless-well-intentioned legislation/regulation-that-has-seriously-unfortunate-yet-doubtless-unintended-consequences.”  That’s about as tactful as you can get.

Alas, Dracula does rise from the grave,*** because I thought I heard noises at a recent Department of Homeland Security event that NISTIR 7622 was going to move beyond “good advice” and morph into a special publication. (“Run for your lives, store up garlic and don’t go out after dark without a cross!”)  The current version of NISTIR 7622 – after two rounds of edits and heaven knows how many thousands of hours of scrutiny – is still unworkable, overscoped and completely unclear: you have a better chance of reading Linear B**** than understanding this document (and for those who don’t already know, Linear B is not merely “all Greek to me” – it’s actually all Greek to anybody).  Ergo, NISTIR 7622 needs to die the true death: the last thing anyone should do with it is make a special publication out of it. It’s doubling down on dreck. Make it stop. Now. Please.


The last section is, to be fair, not really about NIST per se. NIST has been tasked, by virtue of a recent White House Executive Order, with developing a framework for improving cybersecurity. As part of that tasking, NIST has published a Request For Information (RFI) seeking industry input on said framework. NIST has also scheduled several meetings to actively draw in thoughts and comments from those outside NIST. As a general rule, and NISTIR 7622 notwithstanding, NIST is very good at eliciting and incorporating feedback from a broad swath of stakeholders. It’s one of their strengths and one of the things I like about them. More importantly, I give major kudos to NIST and its Director Pat Gallagher for forcefully making the point that NIST would not interfere with IT design, development and manufacture, in the speech he gave when he kicked off NIST’s work on the Framework: “the Framework must be technology neutral and it must enable critical infrastructure sectors to benefit from a competitive [technology] market. (…) In other words, we will not be seeking to tell industry how to build your products or how to run your business.”

The RFI responses are posted publicly and are, well, all over the map.  What is concerning to me is the apparent desire of some respondents to have the government tell industry how to run their businesses. More specifically, how to build software, how to manage supply chain risk, and so forth. No, no, and no. (Maybe some of the respondents are consultants lobbying the government to require businesses to hire these consultants to comply with this or that mandate.)

For one thing, “security by design” concepts have already been working their way into development for a number of years: many companies are now staking their reputations on the security of their products and services. Market forces are working. Also, it’s a good time to remind people that more transparency is reasonable – for example, to enable purchasers to make better risk-based acquisition decisions – but when you buy COTS you don’t get to tell the provider how to build it. That’s called “custom code” or “custom development.” Just like, I don’t get to walk into <insert name of low-end clothing retailer here> and tell them, that I expect my “standard off-the-shelf blue jeans” to ex post facto be tailored to me specifically, made of “organic, local and sustainable cotton” (leaving aside the fact that nobody grows cotton in Idaho), oh, and embroidered with not merely rhinestones but diamonds.  The retailer’s response should be “pound sand/good luck with that.” It’s one thing to ask your vendor “tell me what you did to build security into this product” and “tell me how you help mitigate counterfeiting” but something else for a non-manufacturing entity – the government – to dictate exactly how industry should build products and manage risk. Do we really want the government telling industry how to build products? Further, do we really want a US-specific set of requirements for how to build products for a global marketplace? What’s good for the (US) goose is good for the (European/Brazilian/Chinese/Russian/Indian/Korean/name your foreign country) gander.

An illustrative set of published responses to the NIST RFI – and my response to the response – follows:

1. “NIST should likewise recognize that Information Technology (IT) products and services play a critical role in addressing cybersecurity vulnerabilities, and their exclusion from the Framework will leave many critical issues unaddressed.”

Comment: COTS is general purpose software and not built for all threat environments. If I take my regular old longboard and attempt to surf Maverick’s on a 30 foot day and “eat it,” as I surely will, not merely because of my lack of preparation for 30-foot waves but because you need, as every surfer knows, a “rhino chaser” or “elephant gun” board for those conditions, is it the longboard shaper’s fault? Heck, no. No surfboard is designed for all surf conditions; neither is COTS designed for all threat environments. Are we going to insist on products designed for one-size-fits-all threat conditions? If so, we will all, collectively, “wipe out.” (Can’t surf small waves well on a rhino chaser. Can’t walk the board on one, either.)

Nobody agrees on what, precisely, constitutes critical infrastructure. Believe it or not, some governments appear to believe that social media should be part of critical national infrastructure. (Clearly, the World As We Know It will come to an end if I can’t post a picture of my dog Koa on Facebook.) And even if certain critical infrastructure functions – say, power generation – depend on COTS hardware and software, the surest way to weaken their security is to apply an inflexible and country-specific regulatory framework to that COTS hardware and software. We have an existing standard for the evaluation of COTS IT, it’s called the Common Criteria (see below): let’s use it rather than reinvent the digital wheel.  

2. “Software that is purchased or built by critical infrastructure operators should have a reasonable protective measures applied during the software development process.”

Comment: Thus introducing an entirely new and undefined term into the assurance lexicon: “protective measures.” I’ve worked in security – actually, the security of product development – for 20 years and I have no idea what this means. Does it mean that every product should self defend? I confess, I rather like the idea of applying the Marine Corps ethos – “every Marine a rifleman” – to commercial software. Every product should understand when it is under attack and every product should self-defend. It is a great concept but we do not, as an industry, know how to do that - yet. Does “protective measures” mean “quality measures?” Does it mean “standard assurance measures?” Nobody knows. And any term that is this nebulous will be interpreted by every reader as Something Different. 

3. “Ultimately, <Company X> believes that the public-private establishment of baseline security assurance standards for the ICT industry should cover all key components of the end-to-end lifecycle of ICT products, including R&D, product development, procurement, supply chain, pre-installation product evaluation, and trusted delivery/installation, and post-installation updates and servicing.”

Comment:  I can see the religious wars over tip-of-tree vs. waterfall vs. agile development methodologies. There is no single development methodology, there is no single set of assurance practices that will work for every organization (for goodness’ sake, you can’t even find a single vulnerability analysis tool that works well against all code bases).

Too many in government and industry cannot express concerns or problem statements in simple, declarative sentences, if at all. They don’t, therefore, have any business attempting to standardize how all commercial products are built (what problem will this solve, exactly?). Also, if there is an argument for baseline assurance requirements, it certainly can’t be for everything, or are we arguing that “” is critical infrastructure and need to be built to withstand hostile nation state attacks that attempt to steal your brioche recipe if not tips on how to get sugar to caramelize at altitude?

 4. “Application of this technique to the Common Criteria for Information Technology Security Evaluation revealed a number of defects in that standard.  The journal Information and Software Technology will soon publish an article describing our technique and some of the defects we found in the Common Criteria.”

Comment: Nobody ever claimed the Common Criteria was perfect. What it does have going for it is a) it’s an ISO standard and b) by virtue of the Common Criteria Recognition Arrangement (CCRA), evaluating once against the Common Criteria gains you recognition in 20-some other countries. Putting it differently, the quickest way to make security much, much worse is to have a Balkanization of assurance requirements. (Taking a horse and jumping through mauve, pink, and yellow hoops doesn’t make the horse any better, but it does enrich the hoop manufacturers, quite nicely.)  In the security realm, doing the same thing four times doesn’t give you four times the security, it reduces security by four times, as limited (skilled) resource goes to doing the same thing four different ways. If we want better security, improve the Common Criteria and, by the way, major IT vendors and the Common Criteria national schemes – which come from each CCRA member country’s information assurance agency, like the NSA in the US – have been hard at work for the last few years applying their considerable security expertise and resources to do just that. Having state-by-state or country-by-country assurance requirements will make security worse – much, much worse.

 5. “…vendor adoption of industry standard security models.  In addition, we also believe that initiatives to motivate vendors to more uniformly adopt vulnerability and log data categorization, reporting and detection automation ecosystems will be a significant step in ensuring security tools can better detect, report and repair security vulnerabilities.”

Comment: There are so many flaws in this, one hardly knows where to start. There are existing vulnerability “scoring” standards – namely, the Common Vulnerability Scoring System (CVSS), *****  though there are some challenges with it, such as the fact that the value of the data compromised should make a difference in the score: a “breach” of Aunt Gertrude’s Whiskey Sauce Recipe is not, ceteris paribus, as dire as a breach of Personally Identifiable Information (PII) if for no other reason than a company can incur large fines for the latter, far exceeding Aunt Gertrude’s displeasure at the former. Even if she cuts you out of her will.

Also, there is work going on to standardize descriptions of product vulnerabilities (that is, the format and type). However, not all vendors release the exact same amount of information when they announce security vulnerabilities and should not be required to.  Oracle believes that it is not necessary to release either exploit code or the exact type of vulnerability; e.g., buffer overflow, cross-site request forgery (CSRF) or the like because this information does not help customers decide whether to apply a patch or not: it merely enables hackers to break into things faster. Standardize how you refer to particular advisory bulletin elements and make them machine readable? Sure. Insist on dictating business practices (e.g., how much information to release) – heck, no. That’s between a vendor and its customer base. Lastly, security tools cannot, in general “repair” security vulnerabilities – typically, only patch application can do that.

6. “All owners and operators of critical infrastructure face risk from the supply chain. Purchasing hardware and software potentially introduce security risk into the organization. Creation of a voluntary vendor certification program may help drive innovation and better security in the components that are essential to delivery of critical infrastructure services.”

Comment:  The insanity of the following comment astounds: “Purchasing hardware and software potentially introduce security risk into the organization.” News flash: all business involves “risk.” Not doing something is a risk. So, what else is new? Actually, attempting to build everything yourself also involves risk – not being able to find qualified people, the cost (and ability) to maintain a home-grown solution, and so forth. To quote myself again: “Only God created something from nothing: everyone else has a supply chain.”****** In short, everyone purchases something from outside their own organization. Making all purchases into A Supply Chain Risk as opposed to, say, a normal business risk is silly and counterproductive.  It also makes it far less likely that specific, targeted supply chain threats can be addressed at all if “buying something – anything – is a risk” is the threat definition.

At this point, I think I’ve said enough. Maybe too much. Again, I appreciate NIST as an organization and as I said above the direction they have set for the Framework (not to $%*& with IT innovation) is really to their credit. I believe NIST needs to in-source more of their standards/policy development, because it is their core mission and because consultants have every incentive to create perpetual work for themselves (and none whatsoever to be precise and focused). NIST should adopt a less-is-more mantra vis-a-vis security. It is better to ask organizations do a few critical things well than to ask them to do absolutely everything – with not enough resource (which is a collective industry problem and not one likely to be solved any time soon). Lastly, we need to remember that we are a proud nation of innovators. Governments generally don’t do well when they tell industry how to do their core mission – innovate – and, absent a truly compelling public policy argument for so doing, they shouldn’t try. 

*”Nice Work If You Can Get It,” lyrics by Ira Gershwin, music  by George Gershwin. Don’t you just love Gershwin?

** “Let’s Call The Whole Thing Off.” Another gem by George and Ira Gershwin.

*** Which reminds me – I really hate the expression “there are no silver bullets.” Of course there are silver bullets. How many vampires and werewolves do you see wandering around?

****Speaking of which, I just finished a fascinating if short read: The Man Who Deciphered Linear B: The Story of Michael Ventris.

*****CVSS is undergoing revision.

****** If you believe the account in Genesis, that is.

Friday Aug 17, 2012

Put Up or Shut Up

One of the (usually) unfortunate concomitants of being a veteran in the cybersecurity space (“veteran” as in, I can remember when everyone called it “information security”) is that you get to hear the same themes over and over again (and solve the same security problems over and over again, only with different protocols).* Not to mention, you experience many technical revival meetings, which is industry’s way of promoting the same old same old under new exhortations (“Praise the Lord! I found eternal life with <insert sexy technology cult du jour>!”)

One of the topics that I am tired of talking about and would like us collectively to do something about is (drum roll) information sharing. Now, information sharing is not a cure-all for every ill in cybersecurity. It is a means to an
end, not an end in itself. Specifically, information sharing is a means to enhance situational awareness, which in turn helps networked entities defend themselves better (“Excuse me, I see a mugger is about to swipe your purse. You might want to hit him with it or switch it to your other shoulder.”)

As a basic enabler of better defense, information sharing is certainly a no-brainer, and yet it largely doesn’t happen, or doesn’t happen enough, at least among the good guys. The bad guys, of course, are really good at information sharing. Techniques, tools, top ten lists of badly secured web sites – bring it on, woo hoo. The hacker toolkits are so good now that even someone as technically challenged as I am could probably become a competent Internet evildoer (not that I have any plans to do so). And yet industry and government have spent more time writing tomes, doing PPTs and drafting policy papers that use the magic words “public-private partnership” than making actual – make that “almost any” – progress. Sharing policy papers, I hasten to add, is not the kind of information sharing that solves actual problems. So here it is, all y’all: time to put up or shut up on information sharing.

I say this in my experience as a member of the IT industry Information Sharing and Analysis Center (IT-ISAC) (OK, I am the current president, but I am not speaking for the IT-ISAC) and as a security weenie at Oracle. I can state pretty categorically that I have been astonished – and depressed – at what currently passes for information sharing, despite years of gum flapping about it. The government agencies that are tasked with it generally don’t do it, for example. I find it ironic that the same entities that can’t or won’t tell you you are being broken into – or are about to be – think in some cases that the better solution is for them to just take over protection of your company’s networks after you’ve being broken into. Huh?

More to the point, surprisingly, and delightedly, other agencies that are not tasked with information sharing (e.g., an entity I cannot name by name but that is not part of the Department of Homeland Security (DHS)) recently went to great lengths to contact the IT-ISAC and bring “interesting information” to the attention of the IT-ISAC because they’d seen suspicious activity related to some member companies. Bravo Zulu to you, Unnamed Government Entity. It was not your mission to share that information, but you made an effort and did it, anyway. I wish you'd make a hostile takeover attempt on the entity that is supposed to share information and doesn’t, probably because their lawyers are still mulling it over. If I sound harsh, consider that I have spent 10 years having the exact same conversations over and over and over and nothing seems to change except the people you are having the conversations with. To quote Yoda, “Do or do not. There is no try.”

Other government agencies may call you but you get mysterious intimations and in some cases nothing actionable. I certainly understand that a recipient doesn’t – and probably shouldn’t – receive information about how the reporter got the information (e.g., sources and methods). I know I don’t have a “need to know.” But the information has to be actionable or it’s useless. For example (and I know they meant well), I once got a phone call from Agency X who said, “we have a credible threat that an entity in Country Y (and We All Know Who That Is) is interested in stealing (only they used a more bureaucratic term) the source code for Oracle Product Foo.” Gosh, really? The only news there would be if that country were not out to rip off…er…steal…er…conduct industrial espionage…er…enhance their native manufacturing capacity by ‘active acquisition’… of someone else’s core intellectual property. The next statement was even less helpful: “The details about the threat are classified.” On the one hand, glad Agency X called. Points for trying. On the other hand, the warning was so vague it was not actionable and it certainly didn’t tell me anything I didn’t already know. I wish they’d saved the 35 cents that the call cost and used it to reduce our national debt.

So, the agencies that should share information don’t share much if anything and ones that do in some cases don’t give you information in enough detail such that you can do anything with it. And other good agencies do the right thing although they aren’t tasked with it. It’s not a great report card for the government (more on industry below, lest anyone think I am being one-sided in my criticism). Note that there are people across the political spectrum (and better security really should be an ecumenical issue) who, to their credit, have tried to pass legislation that would help provide “better information sharing” as one of several things we could do to help improve cybersecurity. “Better information sharing” seems a mom-and-secure-apple-pie proposition if ever there was one. Except that a bill that proposed that – and various other iterations of bills – did not pass and for now Congress has gone on vacation like so many of us do in August. There are many reasons why there hasn’t been a consensus cyber bill passed – and I’m not going to go into all that **– but for Pete’s sake, improving government information sharing with industry and vice versa really should be something everyone agrees on.

Another reason that even “kumbaya information sharing 101” couldn’t get a consensus was because of Privacy Concerns. You do wonder about people who are really happy telling intimate details of their lives on Facebook but don’t think the government should be able to receive information about anybody’s attempts to hack critical infrastructure. (Because that’s what we are talking about, not “sending information about the amount of time you spent visiting to the National Security Agency,” which, I am pretty sure, is truly not interested in that information – they have bigger evil fish to fry – and doesn’t view your bunny obsession as a national security threat.)

This is a good time to say that the type of information sharing I am talking about is the voluntary kind (though “highly encouraged” information sharing pursuant to a court order is also good – I’m nothing if not law-abiding). I have zero interest in handing over everything, including the digital kitchen sink, because someone decides they should get everything you have and only then figure out what they actually need. “Need to know” goes for the government, too.

Ergo, at a macro level, I’m glad there are people who are concerned and involved as regards digital privacy. But at the same time, I am frustrated because any time there is even a common sense proposal (legislative or otherwise) about information sharing, privacy hawks seem to come out of the woodwork and Express Grave Concern that either national security or homeland security agencies might actually get useful information from industry to enable them to do their national or homeland security jobs better. Or, God forbid, that industries under non-stop attack from bad guys (including hostile nation states intent on ripping us all off) might actually receive useful and actionable intelligence to help them close open digital doors and windows and keep vermin out. Wouldn’t that be awful?

Because I like analogies, I’d like to offer some perspectives from the real (non-cyber) world that will, at least, illustrate why I am so frustrated and want us to stop talking and start doing. I’d observe that in the physical world, we really don’t seem to have these Concerned Discussions,*** mostly because people understand that we live in communities and that we have a collective interest in making sure we have a secure commons. (Duh, it’s exactly the same issue in the digital world.) Here we go:

Scenario 1: I see a couple walking their dog on the street. They walk by my house and my neighbor’s house. The dog is a Labradope that barks incessantly and the owners don’t clean up after him. ****

Result: I might not like the fact the dog doo-dooed on the arctic willows I painstakingly planted, but this is not a national emergency and it’s not suspicious activity. I’ll clean up after the dog and be done with it. I’m not calling the Wood River Animal Shelter Dog Doo Hotline or the Ketchum Police Department Canine Crap Cop.

Scenario 2: I see someone attempting to enter a window in my neighbor’s house, at 7PM, when my neighbor has gone to the Sun Valley Symphony (they are playing Mahler, whom I don’t care for, which is why I am home instead of at the symphony).

Result: I’m calling the police. I’m also going to give the police as much information as I can about the person doing the B and E (breaking and entering) – what he looks like, how old, how he is dressed, etc. What I am not going to do is think, “Wait, I can’t provide a description of the breaker-inner because gosh, that might violate the perp’s right to privacy and bad taste in clothes. The police showing up when the criminal is doing a breaking and entering job is creating a hostile work environment for him, too.” If you are breaking into someone’s home, you do not have a right to privacy while doing it. Even realizing that there might be false positives (it’s the neighbor's kid, he locked himself out and is breaking into his own house), most of us would rather err on the side of caution and call the cops. We aren’t telling everyone on the planet about “attempted break-in on Alpine Lane,” but we are providing targeted information about a malefactor to the group (Ketchum Police Department) that can do something about it.

In short, if I am a decent neighbor, I should do what I can to protect my neighbor’s house. And as long as I am on the subject, if every house in the neighborhood has been broken into, I would like to know that before someone tries to break into my house. It would be nice if the police told me if there is a rash of B and Es in my neighborhood. (Given it’s a small town in Idaho and we have really good police department, I’m pretty sure they will tell me.)*****

This is what information sharing is, folks. It’s not telling everybody everything whether or not it is interesting or useful. The above examples all have “cyber equivalents” in terms of the difference between sharing “all information” and “sharing interesting information” – which is exactly what we are talking about when we speak of information sharing. There isn’t a neighbor in the world that is busy taping everyone walking dogs by their house (and don’t forget those close-ups of the Labrador committing indiscretions on your plants). Nobody cares about your incontinent Labrador. You share information that is targeted, of value, of interest and where possible, actionable. That’s true in the physical world and in the cyber world.

I’ve been doing a bit of government bashing regarding “failure of government agencies to share information.” Is it only fair that I also do some industry bashing, because information sharing is something some sectors do a lot better than others, yet it is something everyone could and should benefit from. Not to mention, I am mindful of the Biblical wisdom of “Physician, heal thyself” (Luke 4:23).

While the government can add value in information sharing, it is not their job to defend private networks, especially when the private sector – merely by virtue of the fact that they have more digital real estate – gets to see more and thus potentially has more information to share with their neighbors. Not to mention, industry cannot have it both ways. There is a lot of legitimate concern about regulation of cyberspace, mostly because so much regulation has unintended, expensive and often unfortunate consequences. This is all the more reason to Be A Good Cyber Citizen instead of waiting for the government to be the source of all truth or to tell you How To Be A Good Cyber Citizen. Industry participation in information sharing forums is a demonstration of voluntary sector cybersecurity risk management without the force of regulation. As I said earlier, “put up or shut up,” which goes just as much if not more for industry as for government.

While ISACs are not the only information sharing vehicles that exist, they were set up specifically for that purpose (in response to Presidential Decision Directive 63, way back in 1998). It’s a fair cop that some of the ISACs have done better at performing their mission than others. Not all ISACs are equal or even have the same mission. Still, each ISAC has its own examples of success and it is often difficult for those not participating in specific ISACs to see the value they deliver to members (to protect member information that is shared, most ISACs have non-disclosure agreements that prevent information from being shared outside the ISAC membership).

I’d specifically note that the multi-state ISAC and the financial services ISAC both seem to operate very well. There are, I think, many reasons for their success. First of all, the multi-state ISAC and the financial services ISAC have more homogeneity, for lack of a better word. A state is a state is a state – it’s not also a planet. (Except California and Texas, which often seem like Mars to the rest of the country. Bless their lil’ ol’ hearts.) This makes it easier to recognize the obvious benefit of cooperation. To quote Ben Franklin: "We must, indeed, all hang together, or most assuredly we shall all hang separately.” The financial services sector gets this really well: any perceived threat to an individual financial services company is likely to affect all of them, either because of the perception problem that a successful hack creates (“online banking is insecure!”) or because criminals like to repeat successes (to quote Willy Sutton when asked why he robbed banks, “that’s where the money is”). You can’t imagine a bad guy saying, “I’m only going to hack Bank of Foobaria because I don’t like that bank, but Bank of Whateversville is a really nice bank – they hand out dog biscuits – so I am not going to hack them.”

I think leadership is also a factor. I don’t know the originators and past presidents of the Financial Services ISAC, but Bill Nelson has done a tremendous job as the current President of the Financial Services ISAC. I also know Will Pelgrin at the multi-state ISAC and he is a very good, very skilled leader, indeed, and a generous colleague, to boot. Will has been gracious with his time and expertise to me personally in my role as the IT-ISAC president, and I am grateful for it.

While the IT-ISAC has a long list of accomplishments that it is justifiably proud of, the IT-ISAC also faces unique challenges. One of them is the nature of the ISAC and its constituency. The IT industry is less homogeneous than other sectors, including both “soup to nuts” stack vendors as well as security solution vendors that make a business out of sharing threat information. Being a die-hard capitalist, I don’t expect these companies to give away their secret sauce, plus French fries and a hot apple pie to avoid Ben Franklin’s collective hanging. While I think the diversity of the IT sector, the variance in business practices and the “not giving away the store” issues are real challenges to the IT-ISAC, they also provide real benefits. The IT-ISAC provides a forum for bringing together subject matter experts from diverse companies to engage on and discuss common security threats. The IT-ISAC is also moving from an organization focused on vendor vulnerabilities to one that assists members in understanding the rapidly-changing threat environment. For example, we have established a group within the IT-ISAC membership that has agreed to share threat indicator information with each other.

As President of the IT-ISAC, I am committed to doing what I can to try to expand membership, to find common ground (e.g., threat information that even security vendors feel comfortable sharing that benefits everyone, without expecting them to share secret sauce recipes), and finding ways to work with our counterparts in the public sector. I am not the first, and won’t be the last, IT-ISAC president, and I am blessed with an extremely capable executive director and with the generosity of colleagues on the Board. As I learned in my Navy days, I must do my best to steer a steady course to favorable shores.

Lastly, I think the biggest hurdle we in industry collectively need to get over is the trust issue. We seem to be more fearful of other companies than we are of being hacked by bad guys. (“If I share this information, will a competitor use it against me?”) Trust has to be earned, but it can be garnered by outreach and by making an effort to start somewhere. I think of a fine gentleman and public servant who has recently retired from NSA, Tony Sager. Tony was a public face of NSA in terms of working with industry in the information assurance directorate (IAD). He and his team did a lot of outreach: here’s who we are, here’s what we do, let’s talk. Tony did a lot of listening, too. I have said often that if I had a problem in a particular area, I’d not hesitate to call Tony and his team. They had the creds, they had the smarts, and they had earned – yes, earned – my trust. We in industry, who see most of the threats, who are so often the direct victims of them, should take a cue from Tony. Use our “creds” and our intelligence (of all types) to improve the commons. We can start by sharing useful, actionable, valuable information that will help all of us be more secure. It is often said the bad guys are a step ahead of the defenders. This is true with information sharing as well: the bad guys play nicely with other bad guys – so why can’t we good guys get along?

If you are sitting on the sidelines, it is time to get involved and engaged. Instead of sitting on the outside complaining that there is no effective way to share information, join an information sharing organization (I’m partial to ISACs), get involved in it, and help shape and move the organization so that it meets your needs. Just get on with it, already!

* The fact that technology changes but stupidity repeats endlessly is job security for security weenies. Rule number 1 of  nformation security is “never trust any unverified data from a client.” Rule 2 is “see rule 1.” Most security defects stem from failure to heed rule 1 – and we keep doing it every time we introduce new clients or new protocols. The excuse for lazy-ass servers or middle tiers is always, “Gosh, it’s just so much easier to accept any old thing the client hands you because it is computationally intensive to verify it. And nobody would send evil data to a middle tier, would
they?” Right. Just like, think of all the cycles we’d save if we didn’t verify passwords. I’m sure if a client says he is John Doe, he IS John Doe! (Good luck with that.)

** Ok, I lied. One of the reasons various bills failed is because the bill drafters wanted “better security to protect critical infrastructure” but could not actually define “critical infrastructure.” If “it” is important enough to legislate, “it” should be clearly defined in the language of the bill, instead of subject to interpretation (and vast scope increase ex post facto). Just my opinion.

*** With the prospect of increased drone use in our domestic environs, we are going to have a lot more privacy discussions. What I barbecue in my backyard is none of anyone else’s goldurn business.

**** Ok, I know a lot of people love Labs. Apologies to anybody I offended.

***** Since I live a couple of blocks from the police, it’s pretty darn stupid of anybody to try to break into any house in the neighborhood.

Tuesday Jul 17, 2012

Summer Potpourri

Its summer in Idaho, and how! (It was over 90 yesterday and people can say “it’s a dry heat” until the cows come home with heatstroke. My oven is dry, but I don’t want to sit in that, either.) The air is scented with sagebrush (the quintessential Idaho smell), the pine needles I am clearing from my side yard, the lavender and basil in my garden, and the occasional ozone from imminent thunderstorms. It’s those snippets of scent that perfume long, languid summer days here.  With that, I offer my own “literary potpourri” – thought snippets in security.

Digital Pearl Harbor

In the interests of greater service to my country, I would like to propose a moratorium on the phrase “digital (or “cyber”) Pearl Harbor.”  This particular phrase is beyond tiresome cliché, to the point where it simply does not resonate, if it ever did.  The people who use it for the most part demonstrate an utter lack of understanding of the lessons of Pearl Harbor.  It’s as bad as the abuse of the word “decimate,” the original meaning of which was to kill every  10th person (a form of military discipline used by officers in the Roman Army). Unless you were one of the ten, it hardly constituted “utter annihilation,” the now generally accepted usage of “decimate.”   The users and abusers of the phrase “digital Pearl Harbor” typically mean to say that we are facing the cyber equivalent of:

-- A sneak attack

-- That “destroys the fleet” – leaving something critical to our national defense or national well-being in a state of ruin/devastation

-- With attendant loss of life

Not to go on and on about the Pacific War – not that it isn’t one of my favorite topics - but Pearl Harbor turned out to be a blessing in disguise for the United States.  Note that I am not saying it was a blessing for the 2403 men and women who died in the attack (not to mention those who were wounded). In support of my point:

1) The Japanese attack on Pearl Harbor brought together the United States as almost nothing else could have. On December 6, 1941, there were a lot of isolationists. On December 8, 1941, nary a one.  (For extra credit, Hitler was not obligated to declare war on the United States under the terms of the Tripartite Agreement, yet he did so, which means – as if we needed one – we also had a valid excuse to join the European war.)

2) The Japanese did not help their cause any by – due to a timing snafu – only severing diplomatic relations after the attack had begun, instead of before. Thus, Pearl Harbor wasn’t just a sneak attack, but a sneak attack contrary to diplomatic norms.  

Japan left critical facilities at Pearl Harbor intact, due to their failure to launch another attack wave. Specifically, Japan did not destroy the POL (petroleum, oil and lubricant) facilities at Pearl Harbor. They also did not destroy the shipyards at Pearl Harbor. Had the POL facilities been destroyed, the Pacific Fleet would have had to function out of San Francisco (or another West Coast port) instead of Hawai’i.  As for the dry-dock facilities, most of the ships that were sunk  at Pearl Harbor were ultimately raised, refurbished, and rejoined the fleet.  (Special kudos to the divers who did that work, who do not generally get the credit they deserve for it.) The majority of ships on Battleship Row were down on December 7 -- but far from out. 

4) The attack forced the US to rely on their aircraft carriers. Famously, the US carriers were out to sea during the attack, unfortunately for the Japanese, who truly wanted to sink carriers more than battleships.  Consequently, the US was forced (in the early months of the war) to rely on their carriers, and carriers were the future of war-at-sea.  (The Japanese ship Yamato, which, with her sister Musashi, were the largest battleships ever built, was a notable non-force during the war.) 

Japan was never going to prevail against the industrial might of the United States.  Famously, ADM Isoroku Yamamoto -- who had initially opposed the attack on Pearl Harbor -- said, "I can run wild for six months … after that, I have no expectation of success.” It was almost exactly six months between December 7, 1941 and the battle of Midway (June 4-6, 1942), which was arguably the turning point of the Pacific War.

December 7, 1941 was and always will be “a date that shall live in infamy,” but Pearl Harbor was not the end of the United States.  Far from it.

Thus, the people who refer to “Cyber Pearl Harbor” or “Digital Pearl Harbor”(DPH) are using it in virtually complete ignorance of actual history.  Worse, unless they can substantiate some actual “for instances”  that would encompass the basic “here’s what we mean by that,” they run the risk of becoming the boy who cried cyber wolf.  Specifically, and to return to my earlier points:

-- A sneak attack

With the amount of cyber intrusions, cyber thefts, and so on, would any cyber attack (e.g., on critical infrastructure) really be “a sneak attack” at this point? If we are unprepared, shame on us. But nobody should be surprised. Even my notably technophobic Mom reads the latest and greatest articles on cyber attacks, and let’s face it, there are a lot of them permeating even mainstream media. It’s as if Japan had attacked Seattle, Los Angeles and San Diego and pundits were continuing to muse about whether there would be an attack on Pearl Harbor. Duh, yes!

-- That “destroys the fleet” – leaving something critical to our national defense or national well-being in a state of ruin /devastation

If we know of critical systems that are so fragile or interdependent that we could be ruined if they were “brought down,” for pity’s sake let’s get on with fixing them. For the amount of time pundits spending opining on DPH, they could be working to fix the problem.  Hint: unless Congress has been taken over by techno-savvy aliens and each of the members is now supergeek, they cannot solve this problem via legislation. If a critical system is laid low, what are we going to say? “The answer is – more laws!” Yessirree, I had no interest in protecting the power grid before we were attacked, but golly jeepers, now there’s a law, I suddenly realize my responsibilities. (Didn’t we defeat imperial Japan with – legislation? Yep, we threw laws at the Japanese defenders of Tarawa, Guadalcanal, and Peleliu. Let’s give Marines copies of the Congressional Record instead of M4s.)

With attendant loss of life

It’s not inconceivable that there are systems whose failures could cost lives. So let’s start talking about that (we do not, BTW, have to talk about that in gory detail wherein Joe-Bob Agent-of-a- Hostile-Nation-State now has a blueprint for evil). If it -- loss of life -- cannot be substantiated, then it’s another nail in the coffin of using DPH as an industry scare tactic.

To summarize, I have plenty of concerns about the degree to which people rely on systems that were not designed for their threat environments, and/or that embed a systemic risk (the nature of which is that it is not mitigateable – that’s why it is “systemic” risk).  I also like a catchy sound bite as much as the next person (I believe I was the first person to talk about a Cyber Monroe Doctrine).  But I am sick to death of “DPH” and all its  catchy variants. To those who use it: stop waving your hands, and Put Up or Shut Up – read about Pearl Harbor and either map the digital version to actual events or find another CCT (Cyber Catchy Term) . Cyberpocalypse, Cybergeddon, CyberRapture – there are so many potential terms of gigabit gloom and digi-doom – I am sure we as an industry can do better.

Hand waving only moves hot air around – it’s doesn’t cool anything off.

Security Theater

I recently encountered a “customer expectations management” issue -- which we dealt with pretty quickly -- that reminds me of a Monty Python sketch.  It illustrates the difference between “real security” and “security theater” -- those feel-good, compliance-oriented, “everybody else does this so it must be OK” exercises that typically end in “but we can’t have a security problem -- we checked all the required boxes!”  

Here goes. I was told by a particular unnamed development group that customers requesting a penetration test virtually demanded a penetration test report that showed there were vulnerabilities, otherwise they didn’t believe it was a “real” report.  (Sound of head banging on wall.) I’d laugh if the stakes weren’t so high and the discussion weren’t so absurd.

If the requirement for “security” is “you have to produce a penetration test that shows there are vulnerabilities,” that is an easy problem to solve. I am sure someone can create a software program that randomly selects from, say, the oodles of potential problems outlined in the Common Weakness Enumeration (CWE), and produces a report showing one or more Vulnerabilities Du Jour. Of course, it probably needs to be parameterized (e.g.,  you have to show at least one vulnerability per X thousand lines of code, you specify primary coding language so you can tailor the fake vulnerabilities reported to the actual programming language, etc.).  Rather than waste money using a really good tool (or hiring a really good third party), we can all just run the report generator. Let’s call it BogusBreakIt. “With BogusBreakIt, you can quickly and easily show you have state-of- the-art, non-existent security problems – without the expense of an actual penetration test! Why fix actual problems when customers only want to know you still have them? Now, with new and improved fonts!”

With apologies to the Knights Who Say, “Ni,” all we are missing is a shrubbery (not too expensive). The way this exercise should work is that instead of hiring Cheap and Bad Pen Testers R Us to make your customers feel good about imperfect security (why hire a good one to find everything if the bar is so low?), you do the best you can do, yourselves, then, where apropos, hire a really good pen tester, triage the results, and put an action plan together to address the crappiest problems first. If you provide anything to customers, it should not be gory details of problems you have not fixed yet, it should be a high level synopsis with an accompanying remediation plan.  Any customer who really cares doesn’t want “a report that shows security problems,” but a report that shows the vendor is focused on improving security in priority order – and, of course, actually does so.  

Moral: It’s hard enough working in security without wasting scarce time, money, and people on delivering shrubberies instead of security.


I don’t think anybody can doubt my commitment to assurance – it’s been my focus at Oracle for most of the time I’ve worked in security, going on 20 years (I should note that I joined the company when I was 8). It is with mixed feelings that I say that while I was an early (grandfathered) recipient of the Certified Secure Software Lifecycle Professional (CSSLP) certification, I have just let it lapse. I’m not trying to bash the Information Systems Security Certification Consortium (ISC(2)), the developer and “blessing certifier” of the CSSLP, and certainly not trying to denigrate the value of assurance, but the entire exercise of developing this certification never sat well with me. Part of it is that I think it solves the wrong problem – more on which later. Also, my cynical and probably unfair comment when I heard that ISC(2) was developing the CSSLP certification was that, having saturated the market for Certified Information Systems Security Professionals (CISSPs), they needed a new source of revenue. (You do wonder when you look at the number of business cards with a multiplicity of alphabet soup on them: CISM, CISSP, CSSLP, EIEIO (Ok, I made that last one up).)

I let my CSSLP lapse because of a) laziness and b) the fact that I got nothing from it.  Having a CSSLP made no difference to my work or my “professional standing.” It added no value to me personally, or, more importantly, to assurance at Oracle. I started working in assurance before the CSSLP dreamer-uppers ever thought of Yet Another Certification, and my team (and many others in Oracle) have proceeded to continue to try to improve our practices. ISC(2) had no impact on that. None. I wondered why I was paying X dollars to an organization for them to “bless” work that we were doing anyway, that I was doing, anyway, that did not add one iota to our knowledge or practices?  (To be fair, some of the people who work for me have CSSLPs and have kept them current. I can’t speak for what they think they get out of having a CSSLP.)

We have an EIEIO syndrome in security – so many certifications, and really, is security any better? I don’t know. I don’t know how much difference many of these certifications make, except that job search engines apparently look for them as keywords. Many certifications across various industries are used as barriers to market entry, to the point that some forward- thinking states are repealing these requirements as being anti-competitive. (And really, do we need a certification for interior decorators? Is it that critical that someone know the difference between Rococo and Neoclassical styles? C’mon!) In some areas, certifications probably do make sense. Some of them might be security areas. But it doesn’t do any good if the market is so fragmented and we are adding more certifications just to add them. And it really doesn’t do any good if we have too many of them to the point where the next one is JASC – just another security certification. CSSLP felt like that to me.  

I certainly think assurance is important. I just do not know – and remain really, really unconvinced -- that having an assurance “certification” for individuals has done anything to improve the problem space. As I’ve opined before, I think it would be far, far, better to “bake in” security to the computer science and related degree programs than try to “bolt on” assurance through an ex post facto certification. It’s like an example I have used before: the little Dutch boy, who, in the story, put his fingers in leaks in a dike to prevent a flood. We keep thinking if only we had more little Dutch boys, we can prevent a flood. If we don’t fix the “builders” problem,  we – and all the Dutch boys we are using  to stem the flood – will surely drown.

I am not perfect. As good as my team is -- and they and many others in Oracle are the real builders of our assurance program -- they are not perfect. But I stand on our record, and none of that record was, in my opinion, affected one iota by the presence or absence of CSSLPs among our practitioners.

If I may be indulged:

Old MacDonald had some code


And in that code he had some flaws


With a SQL injection here and an XSS there

Run a scan, fuzz your code

Everywhere a threat model

Old MacDonald fixed his code


Last Bits

I mentioned in an earlier blog, in a truly breathtaking example of self-promotion, that my sister Diane and I (writing as Maddi Davidson) had written and published the first in an IT Industry-based murder mystery series,  Outsourcing Murder. The two exciting bits of news about that are, first of all book 2, Denial of Service, is almost (OK, 90%) done.  Stay tuned. The second bit is that Outsourcing Murder made a summer reading list for geeks.  It’s probably the only time in my life I (or rather, my sister and I) will appear in a list with Kevin Mitnick and Neal Stephenson.

*Ira Gershwin would turn over in his grave – I know I’ll never make it as a lyricist.

Wednesday Mar 28, 2012

Pain Comes Instantly

When I look back at recent blog entries – many of which are not all that current (more on where my available writing time is going later) – I am struck by how many of them focus on public policy or legislative issues instead of, say, the latest nefarious cyberattack or exploit (or everyone’s favorite new pastime: coining terms for the Coming Cyberpocalypse: “digital Pearl Harbor” is so 1941). Speaking of which, I personally hope evil hackers from Malefactoria will someday hack into my bathroom scale – which in a future time will be connected to the Internet because, gosh, wouldn’t it be great to have absolutely everything in your life Internet-enabled? – and recalibrate it so I’m 10 pounds thinner. The horror.

In part, my focus on public policy is due to an admitted limitation of my skill set. I enjoy reading technical articles about exploits and cybersecurity trends, but writing a blog entry on those topics would take more research than I have time for and, quite honestly, doesn’t play to my strengths. The first rule of writing is “write what you know.”

The bigger contributing factor to my recent paucity of blog entries is that more and more of my waking hours are spent engaging in “thrust and parry” activity involving emerging regulations of some sort or other. I’ve opined in earlier blogs about what constitutes good and reasonable public policy so nobody can accuse me of being reflexively anti-regulation. That said, you have so many cycles in the day, and most of us would rather spend it slaying actual dragons than participating in focus groups on whether dragons are really a problem, whether lassoing them (with organic, sustainable and recyclable lassos) is preferable to slaying them – after all, dragons are people, too - and whether we need lasso compliance auditors to make sure lassos are being used correctly and humanely. (A point that seems to evade many rule makers: slaying dragons actually accomplishes something, whereas talking about “approved dragon slaying procedures and requirements” wastes the time of those who are competent to dispatch actual dragons and who were doing so very well without the input of “dragon-slaying theorists.”)

Unfortunately for so many of us who would just get on with doing our day jobs, cybersecurity is rapidly devolving into the “focus groups on dragon dispatching” realm, which actual dragons slayers have little choice but to participate in.

The general trend in cybersecurity is that powers-that-be – which encompasses groups other than just legislators – are often increasingly concerned and therefore feel they need to Do Something About Cybersecurity. Many seem to believe that if only we had the right amount of regulation and oversight, there would be no data breaches: a breach simply must mean Someone Is At Fault and Needs Supervision. (Leaving aside the fact that we have lots of home invasions despite a) guard dogs b) liberal carry permits c) alarm systems d) etc.) Also note that many well-managed and security-aware organizations, like the US Department of Defense, still get hacked.

More specifically, many powers-that-be feel they must direct industry in a multiplicity of ways, up to and including how we actually build and deploy information technology systems. The more prescriptive the requirement, the more regulators or overseers a) can be seen to be doing something b) feel as if they are doing something regardless of whether they are actually doing something useful or cost effective. Note: an unfortunate concomitant of Doing Something is that often the cure is worse than the ailment. That is, doing what overseers want creates unfortunate byproducts that they either didn’t foresee or worse, don’t care about. After all, the logic goes, we Did Something.

Prescriptive practice in the IT industry is problematic for a number of reasons. For a start, prescriptive guidance is really only appropriate if:

• It is cost effective
• It is “current” (meaning, the guidance doesn’t require the use of the technical equivalent of buggy whips long after horse-drawn transportation has become passé)*
• It is practical (that is, pragmatic, proven and effective in the real world, not theoretical and unproven)
• It solves the right problem

With the above in mind, heading up the list of “you must be joking” regulations are recent disturbing developments in the Payment Card Industry (PCI) world. I’d like to give PCI kahunas the benefit of the doubt about their intentions, except that efforts by Oracle among others to make them aware of “unfortunate side effects of your requirements” – which is as tactful I can be for reasons that I believe will become obvious below - have gone, to-date, unanswered and more importantly, unchanged.

A little background on PCI before I get too wound up. In 2008, the Payment Card Industry (PCI) Security Standards Council (SSC) introduced the Payment Application Data Security Standard (PA-DSS). That standard requires vendors of payment applications to ensure that their products implement specific requirements and undergo security assessment procedures. In order to have an application listed as a Validated Payment Application (VPA) and available for use by merchants, software vendors are required to execute the PCI Payment Application Vendor Release Agreement (VRA). (Are you still with me through all the acronyms?)

Beginning in August 2010, the VRA imposed new obligations on vendors that are extraordinary and extraordinarily bad, short-sighted and unworkable. Specifically, PCI requires vendors to disclose (dare we say “tell all?”) to PCI any known security vulnerabilities and associated security breaches involving VPAs. ASAP. Think about the impact of that. PCI is asking a vendor to disclose to them:

• Specific details of security vulnerabilities

• Including exploit information or technical details of the vulnerability

• Whether or not there is any mitigation available (as in a patch)

PCI, in turn, has the right to blab about any and all of the above – specifically, to distribute all the gory details of what is disclosed - to the PCI SSC, qualified security assessors (QSAs), and any affiliate or agent or adviser of those entities, who are in turn permitted to share it with their respective affiliates, agents, employees, contractors, merchants, processors, service providers and other business partners. This assorted crew can’t be more than, oh, hundreds of thousands of entities. Does anybody believe that several hundred thousand people can keep a secret? Or that several hundred thousand people are all equally trustworthy? Or that not one of the people getting all that information would blab vulnerability details to a bad guy, even by accident? Or be a bad guy who uses the information to break into systems? (Wait, was that the Easter Bunny that just hopped by? Bringing world peace, no doubt.) Sarcasm aside, common sense tells us that telling lots of people a secret is guaranteed to “unsecret” the secret.

Notably, being provided details of a vulnerability (without a patch) is of little or no use to companies running the affected application. Few users have the technological sophistication to create a workaround, and even if they do, most workarounds break some other functionality in the application or surrounding environment. Also, given the differences among corporate implementations of any application, it is highly unlikely that a single workaround is going to work for all corporate users. So until a patch is developed by the vendor, users remain at risk of exploit: even more so if the details of vulnerability have been widely shared. Sharing that information widely before a patch is available therefore does not help users, and instead helps only those wanting to exploit known security bugs. There’s a shocker for you.

Furthermore, we already know that insider information about security vulnerabilities inevitably leaks, which is why most vendors closely hold such information and limit dissemination until a patch is available (and frequently limit dissemination of technical details even with the release of a patch). That’s the industry norm, not that PCI seems to realize or acknowledge that. Why would anybody release a bunch of highly technical exploit information to a cast of thousands, whose only “vetting” is that they are members of a PCI consortium?

Oracle has had personal experience with this problem, which is one reason why information on security vulnerabilities at Oracle is “need to know” (we use our own row level access control to limit access to security bugs in our bug database, and thus less than 1% of development has access to this information), and we don’t provide some customers with more information than others or with vulnerability information and/or patches earlier than others. Failure to remember “insider information always leaks” creates problems in the general case, and has created problems for us specifically.

A number of years ago, one of the UK intelligence agencies had information about a non-public security vulnerability in an Oracle product that they circulated among other UK and Commonwealth defense and intelligence entities. Nobody, it should be pointed out, bothered to report the problem to Oracle, even though only Oracle could produce a patch. The vulnerability was finally reported to Oracle by (drum roll) a US-based commercial company, to whom the information had leaked. (Note: every time I tell this story, the MI-whatever agency that created the problem gets a bit shirty with us. I know they meant well and have improved their vulnerability handling/sharing processes but, dudes, next time you find an Oracle vulnerability, try reporting it to us first before blabbing to lots of people who can’t actually fix the problem. Thank you!)

Getting back to PCI: clearly, these new disclosure obligations increase the risk of exploitation of a vulnerability in a VPA and thus, of misappropriation of payment card data and customer information that a VPA processes, stores or transmits. It stands to reason that VRA’s current requirement for the widespread distribution of security vulnerability exploit details -- at any time, but particularly before a vendor can issue a patch or a workaround -- is very poor public policy. It effectively publicizes information of great value to potential attackers while not providing compensating benefits - actually, any benefits - to payment card merchants or consumers. In fact, it magnifies the risk to payment card merchants and consumers. The risk is most prominent in the time before a patch has been released, since customers often have little option but to continue using an application or system despite the risks. However, the risk is not limited to the time before a patch is issued: customers often need days, or weeks, to apply patches to systems, based upon the complexity of the issue and dependence on surrounding programs. Rather than decreasing the available window of exploit, this requirement increases the available window of exploit, both as to time available to exploit a vulnerability and the ease with which it can be exploited. Also, why would hackers focus on finding new vulnerabilities to exploit if they can get “EZHack” handed to them in such a manner: a) a vulnerability b) in a payment application c) with exploit code: the “Hacking Trifecta!“ It’s fair to say that this is probably the exact opposite of what PCI – or any of us – would want.

Established industry practice concerning vulnerability handling avoids the risks created by the VRA’s vulnerability disclosure requirements. Specifically, the norm is not to release information about a security bug until the associated patch (or a pretty darn good workaround) has been issued. Once a patch is available, the notice to the user community is a high-level communication discussing the product at issue, the level of risk associated with the vulnerability, and how to apply the patch. The notices do not include either the specific customers affected by the vulnerability or forensic reports with maps of the exploit (both of which are required by the current VRA). In this way, customers have the tools they need to prioritize patching and to help prevent an attack, and the information released does not increase the risk of exploit.

Furthermore, many vendors already use industry standards for vulnerability description: Common Vulnerability Enumeration (CVE) and Common Vulnerability Scoring System (CVSS). CVE helps ensure that customers know which particular issues a patch addresses and CVSS helps customers determine how severe a vulnerability is on a relative scale. Industry already provides the tools customers need to know what the patch contains and how bad the problem is that the patch remediates.

So, what’s a poor vendor to do? Oracle is reaching out to other vendors subject to PCI and attempting to enlist then in a broad effort to engage PCI in rethinking (that is, eradicating) these requirements. I would therefore urge all who care about this issue, but especially those in the vendor community whose applications are subject to PCI and who may not have know they were being asked to tell-all to PCI and put their customers at risk, to do one of the following:

• Contact PCI with your concerns
• Contact Oracle (we are looking for vendors to sign our statement of concern)
• And make sure you tell your customers that you have to rat them out to PCI if there is a breach involving the payment application

I like to be charitable and say “PCI meant well” but in as important a public policy issue as what you disclose about vulnerabilities, to whom and when, meaning well isn’t enough. We need to do well. PCI, as regards this particular issue, has not done well, and has compounded the error by thus far being nonresponsive to those of us who have labored mightily to try to explain why they might want to rethink telling the entire planet about security problems with no solutions.

By Way of Explanation…

Non-related to PCI whatsoever, and the explanation for why I have not been blogging a lot recently, I have been working on Other Writing Venues with my sister Diane (who has also worked in the tech sector, inflicting upgrades on unsuspecting and largely ungrateful end users). I am pleased to note that we have recently (self-)published the first in the Miss Information Technology Murder Mystery series, Outsourcing Murder. The genre might best be described as “chick lit meets geek scene.”

Our sisterly nom de plume is Maddi Davidson and (shameless plug follows): you can order the paper version of the book on Amazon, or the Kindle or Nook versions on or, respectively. From our book jacket:

Emma Jones, a 20-something IT consultant, is working on an outsourcing project at Tahiti Tacos, a restaurant chain offering Polynexican cuisine: refried poi, anyone? Emma despises her boss Padmanabh, a brilliant but arrogant partner in GD Consulting. When Emma discovers His-Royal-Padness’s body (verdict: death by cricket bat), she becomes a suspect.With her overprotective family and her best friend Stacey providing endless support and advice, Emma stumbles her way through an investigation of Padmanabh’s murder, bolstered by fusion food feeding frenzies, endless cups of frou-frou coffee and serious surfing sessions. While Stacey knows a PI who owes her a favor, landlady Magda urges Emma to tart up her underwear drawer before the next cute cop with a search warrant arrives. Emma’s mother offers to fix her up with a PhD student at Berkeley and showers her with self-defense gizmos while her old lover Keoni beckons from Hawai’i. And everyone, even Shaun the barista, knows a good lawyer.

Book 2, Denial of Service, is coming out this summer.

* Given the rate of change in technology, today’s “thou shalts” are easily next year’s “buggy whip guidance.”

Wednesday Aug 24, 2011

Those Who Can’t Do, Audit

I am often asked what the biggest change is that I’ve seen in my years working in information security. (Most of those who ask are too polite to ask how many years I’ve worked in information security. I used to say I joined Oracle when I was 8. Now, I tell everyone I started at Oracle when I was 3. Pretty soon, I’ll be a prenatal employee.)

There are too many changes to pick just one, but on a personal level, I seem to spend a lot more time on government affairs (relative to the rest of my job) that I used to: working with others in industry to discuss cybersecurity-related public policy, meeting with staffers on the Hill, commenting on draft legislation, commenting on others’ comments on draft legislation, and so on. On the one hand, it’s a natural result of the amount of dependence that everyone has on cybersecurity and the amount of highly publicized breaches, hacks, and so on we continue to see. You can hardly open the paper – or open an email with a link to the daily electronic paper – without reading about another data breach, “outing” of sensitive information, new thing you didn’t even know was IP-accessible being broken into (e.g., starting cars remotely), and so on.

Legislators often want to Do Something when they see a problem – that’s why we elected them, presumably. I’ve opined in previous blogs on the importance of defining what problem you want to solve, specifying what “it” is that you want to legislate, understanding costs – especially those pertaining to unintended consequences - and so on. I spend more time on govie affairs now because more legislators are proposing legislation that they hope will improve cybersecurity.

Most people who help frame public policy discussions about cybersecurity are well intended and they want to minimize or resolve the problem. However, there are also a whole lotta entities that want to use those discussions to get handed a nice, big, fat “public policy printing press”: ideally, legislative mandates where their businesses receive a direct and lucrative benefit, ka-ching. Their idea of public policy begins and ends at the cash register.

Some businesses would love to be in a position where they could write/draft/influence legislation about how no commercial company can be trusted with security and therefore needs someone (presumably people like themselves) to bless their systems and all their business practices – the more, the merrier. This includes legislative mandates on suppliers – who, as we all know – are busy throwing crappy code over the wall with malice aforethought. Those evil suppliers simply cannot be trusted, and thus the entirety of supplier business practices in security must be validated by Outside Experts* if not the entirety of their code scanned. (Nice work if you can get it.) (Aside: boy, with all the energy we suppliers expend on deliberately producing rotten code and putting our customers’ systems at risk – not to mention our own businesses since we run on our own stuff - it’s really hard to find time for other malicious acts like starving baby rabbits and kitties.**)

At the same time, these businesses are less than forthcoming when asked how they will be vetted: what do they find, how well do they find it and at what cost, since everybody who has worked with code scanning tools know they need to be tuned. (Having to plow through 1000 alleged vulnerabilities to find the three that are legitimate is way too expensive for any company to contemplate doing it.) Now is a perfect time for a disclosure: I am on an advisory board to a company in this sector.

One company in particular is beginning to get under my skin on a lot of levels pertaining to “creating a market for themselves.” Let’s call them SASO (Static Analysis Service Offering) that – ta da! - do static analysis as a service offering. More specifically, they analyze the binaries to do static analysis – an important point. When SASO got started, I thought they had a useful value proposition in that a lot of small software companies don’t have security expertise in-house – one reason why Oracle has strong “business alignment” processes when we acquire a company that include alignment with our assurance practices, which the acquired entities are by-and-large really happy to do. The small company wants to increase their ability to sell into core markets and to do so they may need to convince prospective customers that their code is appropriately secure. They hire SASO to analyze their code, and SASO produces a summary report that tells the customer whether there are significant potentially unfixed vulnerabilities and also gives the small company the details of those vulnerabilities so they can validate actual vulnerabilities and then fix them. Fair enough, and a good service offering, to a point, more on which later. The real benefit to the small company is to get actionable intelligence on problems in their code earlier because for one, it’s almost always easier and cheaper to fix these before products ship, and/or remediate them in some severity order. Also, it’s not always easy to use static analysis tools, yourself: the tools require knowledgeable people to run them and analyze the results.

It’s not, in other words, the “gold star” from SASO they get that has value; it’s the SASO expertise they get to fix issues earlier until they get that expertise in-house and up-to-speed. And they really do need that expertise in-house because security is a core element of software development and you cannot outsource it -- or you will have no security. (Among other things Oracle does in terms of secure development, we have our ethical hacking team conduct product assessments. They are as good as anybody in industry – or better – and they find problems that code analysis tools do not find and never will find. Their expertise is often codified in our own tools that we use in addition to commercially available static and dynamic analysis tools.)

Of course, the market for “helping small inexperienced companies improve the security-worthiness of their code” isn’t big enough, so SASO has been trying to expand the size of their market by means of two understandably - yet obnoxiously - self-promoting activities. One of them is by convincing entire market sectors that they cannot trust their vendors, and their suppliers’ code needs to be “tested” by SASO. You do want to ask, why would anybody trust them? I mean, more and more of SASO’s potential market size is based on FUD. Whereas, the suppliers’ market is based on their customers’ ability to run their entire businesses on the software or hardware. That is, customers bet the digital farm on their suppliers, and the suppliers understand this. If you lose your customer’s confidence for whatever reason – security among them – your customers will switch suppliers faster than you can say “don’t bother with the code scan.” And thus, suppliers are out of business if they screw it up, because their competitors will be ruthless. Competitors are ruthless.

Whom do you think is more trustworthy? Who has a greater incentive to do the job right – someone who builds something, or someone who builds FUD around what others build? Did I mention that most large hardware and software companies run their own businesses on their own products so if there’s a problem, they – or rather, we – are the first ones to suffer? Can SASO say that? I thought not.

Being approached by more and more companies who want our code SASOed has led me to the “enough, already” phase of dealing with them. For the record, we do not and will not allow our code to be SASOed; in fact, we do not allow any third parties access to source code for purposes of security analysis (with one exception noted below). I’ve also produced a canned response that explains to customers why SASO will never darken our code base. It covers the following points, though admittedly when I express them to customers they are less glib and more professional:

1) We have source code and we do our own static analysis. Ergo, I don’t need SASO or anybody else to analyze our binaries (to get at the source code). Moreover, we have not only one but two static analysis tools we use in-house, one of which we built (Parfait, which is built by Sun Labs, a fine outfit we got via the Sun acquisition). Also, as I’ve opined at length in past blog entries, these tools are not “plug and play” – if they were, everybody would run down to Radio Shack, or wait for late night television to see advertisements for Code Scannerama (“finds all security vulnerabilities and malware in 28 seconds -- with No False Positives! But wait, there’s more: order now, and get a free turnip twaddler!”). For the amount of time we’d expend to work with SASO and get useful, accurate and actionable output – we could just get on with it and use the tools we’ve already licensed or already have. SASOing our code hurts us and doesn’t help us.

2) Security != testing. I am all for the use of automated tools to find vulnerabilities in software – it’s a code hygiene thing – but it is not the totality of assurance. Oracle has an extensive assurance program for our products that includes the use of static analysis tools. (In fact, Oracle requires the use of multiple tools, some of which we have developed in-house because no single tool “does it all.”) We also track compliance with our assurance program and report it to our senior management, all the way up to the CEO. In short, we do a far more comprehensive job in security than SASO can do for us. (I also note that – hype to the contrary – these tools will not find much if any malware in code. I am a blond and a bad programmer and it took me about ten minutes to think of ways to put bad stuff in code in a way that would not be detectable by automated code analysis.) “Code analysis” may be a necessary element for security-worthy code but it sure ain’t sufficient.

3) Precedent. It’s a really, really bad precedent to hand your source code to a third party for purposes of “security analysis” because gee, lots of governments have asked for the same privilege. And some governments who ask (I won’t name them, but we all know who they are) engage in state-sponsored industrial espionage so you might as well just kiss your intellectual property good bye as hand your code to those guys (“Wanna start a new software/hardware company? Here’s our source code, O Official Government of Foobaria IP Theft Department!”). Some governments also want to easily do security analysis they then exploit for national security purposes – e.g., scan source code to find problems that they hand to their intelligence agencies to . Companies can’t really say, “we will allow SASO to scan our code if customers nudge us enough” and then “just say no” to governments who want exactly the same privilege. Also, does anybody think it is a good idea for any third party to amass a database of unfixed vulnerabilities in a bunch of products? How are they going to protect that? Does anybody think that would be a really nice, fat, juicy hacker target? You betcha.

(As a matter of fact, Oracle does think that “bug repositories” are potentially nice, fat, juicy hacker targets. Security bug information is highly sensitive information, which is why security bugs are not published to customers and our bug database enforces “need to know” using our own row level access control technology. The security analysts who work for me have security bug access so they can analyze, triage, ensure complete fixes, and so on, but I don’t, because we enforce “need to know,” and I don’t.) I don’t trust any third party to secure detailed bug information and it is bad public policy to allow a third party to amass it in the first place.

4) Fixability. Only Oracle can fix vulnerabilities: SASO cannot. We have the code: they don’t.

5) Equality as public policy. Favoring a single customer or group of customers over others in terms of security vulnerability information is bad public policy. Frankly, all customers' data is precious to them and who would not want to be on the "insider" list? When we do have vulnerabilities in products, all customers get the same information at the same time and the same level of information. It’s basic equity and fairness. It’s also a lot better policy than playing favorites.

6) Global practices for global markets. The industry standard for "security evaluations" is an ISO standard called the Common Criteria (ISO 15408) that looks at the totality of security functionality needed to address specific threats as well as "assurance." A product with no exploitable security defects - that has no authentication, auditing or access control – and isn’t designed to protect against any threats - is absolutely, utterly useless. SASOing your code does not provide any information about the presence or absence of security functionality that meets particular threats. It’s like having someone do a termite inspection on the house you are about to buy but not looking at structural soundness: you know you don’t have bugs – pun intended – but you have not a clue about whether the house will fall down. Common Criteria evaluations do include vulnerability analysis but do not require static analysis. However, many vendors who Common Criteria-evaluate their product also do static analysis, whether or not they get any “credit” for doing it, because it is good practice to do so. Our Common Criteria labs do in some cases get source code access because of the type of analysis they have to do at some assurance levels. But the labs are not “governments,” they are licensed (so there is some quality control around their skill set), access to our source code is incredibly restricted, and we have highly enforceable liability provisions regarding protection of that source code.

7) All tools are not created equal. The only real utility in static analysis is in using the tools in development and fixing what you find in some priority order. In particular, requiring static analysis – e.g., by third parties - will not lead to significant improvement except possibly in the very short term and in the medium to long term will increase consumer costs for no benefit. Specifically: · The state of the art in vulnerability discovery tools is changing very rapidly. Mandating use of a particular tool - or service - will almost guarantee that the mandated tool/service will be obsolete quickly, resulting in added cost for incremental benefit at best. Obsolescence is especially guaranteed if a particular tool or service is mandated because the provider, having been granted a monopoly, has no incentive to improve their service. · The best types of tools used to discover vulnerabilities differ depending on the type of product or service that is being analyzed. Mandating a particular tool is thus suboptimal at best, and an expensive mistake at worst.

I said at the outset that SASO has a useful model for smaller companies that perhaps do not have security expertise in house - yet. But having said that, I offer a cautionary story. A company Oracle acquired some time ago had agreed to have their code SASOed at the request of a customer - before we acquired them. The first problem that created was that the team, by agreeing to a customer request – and continuing to agree to “outside code auditing” until we stopped it – set the market expectation that “gosh, we don’t know what we are doing in security and you shouldn’t trust us,” which is bad enough. What is far, far, worse: I believe this led to a mentality that “those (SASO or other) testing people will find security issues, so I don’t need to worry about security.” I told the product group that they absolutely, positively, needed in-house security expertise, that “outsourcing testing” would create an “outsourcing security” mentality that is unacceptable. Oracle cannot – does not – outsource security.

By way of contrast, consider another company that does static analysis as a service. Let’s call them HuiMaika‘i (Hawaiian for “good group”). HuiMaika‘i provides a service where customers can send in a particular type of code (i.e., based on a particular development framework) and get a report back from HuiMaika‘i that details suspected security vulnerabilities in it. Recently, a HuiMaika‘i customer sent them code which, when analyzed, was found to contain multiple vulnerabilities. HuiMaika‘i somehow determined that this code was actually Oracle code (that is, from an Oracle site) and, because it is their policy not to provide vulnerability reports to customers who submit code not owned by the customer, they returned the vulnerability report to Oracle and not the paying customer. (How cool is that?)

I think their policy is a good one, for multiple reasons. The first is that running such vulnerability-detecting software against code not owned by the customer may violate terms-of-use licenses (possibly resulting in liability for both the customer and the vendor of such vulnerability detecting software). The second reason is that it would be best all around if the vendor of the software being scanned be informed about possible vulnerabilities before public disclosure so that the vendor has a chance to provide fixes in the software. Having said that, you can bet that Oracle had some takeaways from this exercise. First, we pulled the code down wikiwiki (Hawaiian for “really fast”) until we could fix it. And we fixed it wikiwiki. Second, my team is reminding development groups - in words of one syllable - that our coding standards require that any code someone gets from Oracle – even if “just an example” - be held to the same development standards as products are. Code is code. Besides, customers have a habit of taking demo code that does Useful Stuff and using it in production. Ergo, you shouldn’t write crappy code and post it as “an example for customers.” Duh.

Our third takeaway is that we are looking at using HuiMaika‘i onsite in that particular development area. It wasn’t, by the way, just because of what they found - it’s the way they conducted themselves: ethically, professionally, and they didn’t play “vendor gotcha.” Thanks very much, Hui. Small company – big integrity.

Returning to SASO, the other way – besides marketing “you can’t trust suppliers but you can trust us” - in which they are trying to expand their market is a well-loved and unfortunately, often-attempted technique – get the government to do your market expansion work for you. I recently heard that SASO has hired a lobbyist. (I did fact check with them and they stated that, while they had hired a lobbyist, they weren’t “seriously pursuing that angle” – yet.) I have to wonder, what are they going to lobby for? Is it a coincidence that they hired a lobbyist just at the time we have so much draft cybersecurity legislation, e.g., that would have the Department of Homeland Security (DHS) design a supply chain risk management strategy – including third party testing of code - and then use that as part of Federal Acquisition Regulations (FAR) to regulate the supply chain of what the government buys? (Joy.)

I suspect that SASO would love the government to mandate third party code scans – it’s like winning the legislative lottery! As to assurance, many of my arguments above about why we “just say No, to S-A-S-O!” are still equally – if not more – relevant in a legislative context. Secondly, and related to “supply chain” concerns, you really cannot find malware in code by means of static analysis. You can find some coding errors that lead to certain types of security vulnerabilities - exploitable or not. You might be able to find known patterns of “commercially available” malware (e.g., someone downloaded a library and incorporated it into their product, and did not bother to scan it first before doing so. Quelle surprise! The library you got from is infected with a worm). That is not the same thing as a deliberately introduced back door, and code scans will not find those.

In my opinion, neither SASO - nor any other requirement for third party security testing - has any place in a supply chain discussion. If the concern is assurance, the answer is to work within existing international assurance standards, not create a new one. Particularly not a new, US-only requirement to “hand a big fat market win by regulatory fiat to any vendor who lobbied for a provision that expanded their markets.” Ka-ching.

The bigger concern I have is the increasing amount of attempts by many companies, not to innovate, but to convince legislators that people who actually do stuff and build things can’t be trusted (how about asking themselves the same question?). I recently attending a meeting in which one of the leading presenters was a tenured professor of law whose area of focus is government regulation, and who admitted having no expertise in software. Said individual stated that we should license software developers because “we need accountability.” (I note that as a tenured professor, that person has little accountability since she is “unfireable” regardless of how poor a job she does, or whether there is any market demand for what she wants to teach.)

I felt compelled to interject: “Accountable to whom? Because industry hires many or most of these grads. We can enforce accountability – we fire people who don’t perform. If your concern is our accountability to our customers – we suppliers absolutely are accountable. Our customers can fire us – move to a competitor – if we don’t do our jobs well.”

In some cases, there are valid public policy arguments for regulation. The to- and fro- that many of us have in discussions with well-meaning legislators is what the problem really is, whether there is a market failure and - frankly -whether regulation will work, and at what cost. I can agree to disagree with others over where a regulatory line is drawn. But having said that, what I find most distasteful in all the above, is the example of a company with a useful service - “hiring guns to defend yourself, until you get your own guns” - deciding that instead of innovating their way to a larger market, they are FUDing their way to a larger market and potentially “regulating” their way to a larger market.

America has been in the past, a proud nation of innovators, several of whom I have been privileged to know and call “friends.” I fear that we are becoming nation of auditors, where nobody will be able to create, breathe, live, without asking someone else, who has never built anything but has been handed power they did not earn, “Mother, May I?”

Live free, or die.  

* Aside from everything else, I draw the line at someone who knows less than I do about my job tell me how to do it. Unless, of course, there is some reciprocity. I’d like to head the Nuclear Regulatory Commission. Of course, I don’t know a darn thing about nuclear energy, but I’m smart and I mean well. Of course, many people given authority to do things nobody asked them to do - and wouldn’t ask them to do - create havoc because, despite being smart and meaning well, they don’t have a clue. Like the line goes from The Music Man, “you gotta know the territory.”

** Of course we don’t do that – it was a joke. I have a pair of bunnies who live at the end of my street and I’ve become quite attached to them. I always slow down at the end of the street to make sure I don’t hit them. I also hope no predator gets them because they are quite charming. Of course, I might not find them so charming if they were noshing on my arugula, but that’s why you schlep on over to Moss Garden Center and buy anti-deer and anti-bunny spray for your plants. I like kitties, too, for the record.

Book of the Month

Bats at the Library by Brian Lies 
This is a children’s book but it is utterly charming, to the point where I bought a copy just to have one. I highly recommend it for the future little reader in your life. Bats see that a library window has been left open, and they enter the library to enjoy all its delights, including all the books (and of course, playing with the copy machine and the water fountain). The illustrations are wonderful, especially all the riffs on classic children’s stories (e.g.,Winnie the Pooh as a bat, Peter Rabbit with bat wings, the headless horseman with bat wings, etc.). There are two other books in the series: Bats at the Beach and Bats at the Ballgame. I haven’t read them, but I bet they are good, too.

Shattered Sword: The Untold Story of the Battle of Midway by Jonathan Parshall and Anthony Tully
I can’t believe I bought and read, much less I am recommending yet another book on the battle of Midway, but this is a must read. The authors went back to primary Japanese sources and discuss the battle from the Japanese point of view. It’s particularly interesting since so many Western discussions of the battle use as a primary source the account of the battle by Mitsuo Fuchida (who led the air strike on Pearl Harbor and who was at Midway on ADM Nagumo’s flagship), which was apparently largely – oh dear – fabricated to help various participants “save face.” For example, you find out how important little details are such as the US forces regularly drained aviation fuel lines (to avoid catastrophic fires during battles – which the Japanese did not do) and that the Japanese – unlike the US - had dedicated damage control teams (if the damage control team was killed, a Japanese ship could not do even basic damage control). It’s an amazing and fascinating book, especially since the authors are not professional historians but regular folks who have had a lifelong interest in the battle. Terrific read.

Neptune’s Inferno: The US Navy at Guadalcanal by James Hornfischer 
I might as well just recommend every book James Hornfischer has ever written since he is a wonderful naval historian, a compelling storyteller, and incorporates oral history so broadly and vividly into his work. You see, hear, and feel -- to the extent you can without having been there -- what individuals who were in the battle see, heard and felt. Despite having read multiple works on Guadalcanal (this one focuses on the sea battles, not the land battle), I came away with a much better understanding of the “who” and “why.” I was so convinced it was going to be wonderful and I’d want to read it again that I bought it in hardcover (instead of waiting for paperback and/or getting it at the library). Terrif.

Cutting for Stone by Abraham Verghese 
I don’t generally read much contemporary fiction since so much of it is postmodern dreary drivel. However, a friend bought this for me - and what a lovely gift! This is a compelling story (twin brothers born in Ethiopia to a surgeon farther and a nun mother) about what constitutes family - and forgiveness. It has been a best seller (not that that always means much) and highly acclaimed (which in this case does mean much). The story draws you in and the characters are richly drawn. I am not all that interested in surgery but the author makes it interesting, particularly in describing the life and work of surgeons in destitute areas.

Friday Apr 08, 2011

The Bucket List

[Read More]

Thursday Sep 02, 2010

The Root of The Problem

[Read More]

Thursday Jul 15, 2010

Physician, Heal Thyself

[Read More]

Tuesday Sep 08, 2009

Summer R & R

[Read More]

Monday May 11, 2009

What’s Mine Is Mine

[Read More]

Monday Apr 27, 2009

Elk Encounters

[Read More]

Saturday Feb 07, 2009

Loose Dogs

[Read More]



« July 2016