2 November 2002
See Common Criteria for information security Evaluation Assurance Levels (EALs): http://www.commoncriteria.org/docs/EALs.html
Date: Sat, 2 Nov 2002 13:31:42 -0500 From: Adam Shostack <adam@homeport.org> To: Jonathan S. Shapiro <shap@eros-os.org> Cc: cryptography@wasabisystems.com Subject: Re: Windows 2000 declared secure On Sat, Nov 02, 2002 at 11:54:36AM -0500, Jonathan S. Shapiro wrote: | The word "moderate" here is very unfortunate. In reading such | statements, one needs to understand a bit of subtext. The Common | Criteria community is very concerned about the possibility that people | will perceive assurance as impossibly difficult. In consequence, there | has been a tendency to a form of "grade inflation." The effectiveness of | the levels is modestly exaggerated, and the importance of going for | higher levels is grossly understated. | | One unfortunate consequence is that NSA has seen no need to publish | guidelines on performing higher-level evaluations, because there has | been no demand. Could you define 'importance' here? Given a lack of demand, what are you using as criteria? How can we translate that into something that's important to buyers? Or otherwise convince the buyers of systems to demand better? (Leading to NSA publishing those higher level evaluation guidelines, etc.) Adam
Date: Sat, 02 Nov 2002 15:12:51-0500 From: Jonathan S. Shapiro shap@eros-os.org To: Adam Shostack adam@homeport.org Cc: cryptography@wasabisystems.com Subject: Re: Windows 2000 declared secure On Sat, 2002-11-02 at 13:31, Adam Shostack wrote: > Could you define 'importance' here? Given a lack of demand, what are > you using as criteria? How can we translate that into something > that's important to buyers? Or otherwise convince the buyers of > systems to demand better? (Leading to NSA publishing those higher > level evaluation guidelines, etc.) Apologies once again for the length of my reply. The issues are both complicated and political. I also need to preface this by saying that I am an active participant in the dialog around this issue. My usual role is poking sharp sticks into people's eyes, so please read what I have to say skeptically. Context: There are international mutual-recognition treaties covering EAL4 and below, so if you get an EAL4 evaluation in Germany, it's accepted as binding in the US. Above EAL4 there is no mutual recognition. From talking to various assurance evaluators (current and former) and also to people within NSA, the alleged rationale behind this is in two parts: (1) There is a perception that commercial products won't seek higher evaluation, so there has been no real pressure to push the treaties beyond this. (2) There is a perception that products seeking assurance above EAL4 are likely to be targeted primarily at military and/or sensitive applications. No country wants to be in the position of being treaty-obligated to accept assurance from another country about militarily sensitive materials. Given that an EAL4 certification can fairly be characterized as "nowhere near good enough for serious commercial use today", I think it is fair to harshly criticize these rationales as rather thin rationalizations. I am (ahem) cynical. I suspect that the absence of treaty agreement may also have something to do with the fact that as of today the NSA has not published procedural guidelines for anything above EAL4. In fact, they have not finished writing them. Brian Snow (NSA), who has the mandate for publication of such guidelines in the U.S., is in a fairly absurd position. On the one hand he goes around telling everybody "it's important to do higher assurance", but on the other hand the current state of affairs (which is his responsibility) makes this impossible. There is another subtext. One agenda of the evaluation community is to get people in the habit of doing evaluations before they raise the stakes. In order for this strategy to work, products have to pass the evaluations. The de facto effect is a desire not to set too high a bar. I personally disagree with this strategy, but even if I did I would argue that EAL4 is not a barrier to any current commodity operating system, and the US national interest is not served so long as the best standards are inadequate for daily commercial use. World interests aren't served either, but the NSA mandate stops with the US. When pressed, Brian Snow says that until somebody wants to actually do a higher level evaluation, NSA cannot justify the expense of doing the higher level guidelines, but that they could proceed with a higher evaluation today using the older guidelines that were applied under TCSEC. This is probably true, but it is not clear whether such an evaluation would have any commercial value, because the quality of the resulting evaluation is not characterized by a published standard of practice. I am publicly on record as saying that EROS will attempt EAL7 evaluation as soon as possible after NSA published appropriate evaluation guidelines. This is one of the issues that I use sharp sticks on. Brian usually takes them in the eye, and I think is privately glad that somebody is agitating over this issue. More specifically, I have stated publicly that the single greatest impediment to a successful EAL7 evaluation for EROS is the absence of relevant guidelines. Without those guidelines, we can neither sensibly prepare for evaluation nor know what the result would mean. A lot of people out there who do secure OS work know about our stuff and our formal verification work and find these statements quite threatening. They know that (a) EROS can probably do it, and (b) most of them cannot. In fairness, I should add that the conventional approach to this has been for the customer to pay for the evaluation, and Hopkins certainly isn't going to do this. Until this changes, I can pretty much guarantee that no open source OS will go through a high-level evaluation. Brian and I have talked intermittently about how to solve this, and I think we have a strategy worked out for when the time comes. So the current state of affairs is that for levels above EAL4 the evaluation must be performed with participants from the government of the host country, and has no standing outside of the host country. I suspect that someone achieving a successful EAL5 evaluation in the US could claim EAL4 elsewhere under the treaties, but a US-achieved EAL5 would not qualify a product to be submitted on, say, a German military solicitation requiring an EAL5 certification. Note that the current policies have a curious effect. Companies pursue evaluation primarily for the marketing benefit of the certification. The minute they step from EAL4 to EAL5, they suddenly need to spend millions of dollars **per country** and the marketing payoff stops. A consequence is that higher levels of assurance work are not cost effective on the development end. The de facto effect is to ensure that there is a strong economic disincentive to make a truly secure operating system widely available to the public sector. Given the historical export policies of the US, the fact that secure systems are still considered sensitive by the armed services, and the fact that B2 or better systems may still be export controlled under ITAR (their status is ambiguous at present), one must question whether this outcome is an accident or an intentional result of undisclosed policy decisions by various groups within the U.S. government. Whatever policy NSA thinks is a good idea, the impact on the U.S. commercial sector is very clear. Until there is a unified incentive structure resulting in the construction of widely available secure systems, both U.S. and world businesses will continue to be largely bare-ass naked where security is concerned. Further, it follows from this that the U.S. civilian sector is vulnerable to attack in the event of an extended military action. It has long been recognized that the civilian sector is *vital* for provisioning and supply in any extended military action (i.e. longer than one week). In consequence, I would go so far as to argue that the current ITAR policy is actively *contrary* to U.S. interests in both the commercial and military sectors. Paul Karger has gone further, suggesting that "... the current state of civilian sector computer security and the ITAR regulations represents a clear and present danger to the security interests of the United States." For myself, I prefer a slighly less confrontational approach. My answer generally goes something like: Well, we've been operating under the current policy for 35 years, and we're pretty much naked. If you want the civilian sector to be even modestly more secure, perhaps criminalizing the open sale of secure systems isn't the way to go. All other things being equal, it is a legitemate national security goal to prevent our enemies from having this technology. The cost of the current policy is that we are more vulnerable than they are, because we rely utterly on computers in our civilian infrastructure, and most of our enemies do not. The current policy is a knife at our own throats. So finally, Adam was really asking "what do we do about it?" The key, I think, is to recognize that international treaties don't matter. If a reputable group of recognized computer scientists were to publish a well thought out set of evaluation criteria for higher level evaluation, NSA would have little political choice but to adopt them -- perhaps with modification. Similarly, performing a successful evaluation against these criteria using publicly contributed resources would disenfranchise the people who say "real security is too hard". Both activities would have to be done in a way that was very much beyond reproach. In part, this means that it needs to be done in an open source way -- that is, *all* of the process documentation and such needs to be publicly reviewable. Enough said for now, but I will hopefully have more to say on this within the next few months. shap --------------------------------------------------------------------- The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to majordomo@wasabisystems.com