Re: Top Ten Web App Sec Problems

Alex,

My ears were burning. Here are my $0.02.

> Couple of things to note about the paper:

All 45 of the engagements were revenue-generating engagements. To extent that there is sample bias, it is fair to say the sample is skewed towards companies that are better off than most (that is, they had the presence of mind to hire us). So, your presumption is probably correct.

>
> * tools are downplayed in the analysis, yet no hard numbers are

Fair enough. What I was really driving at is that, in the end, what we reported as findings were ultimately things that were significant enough to percolate up through the mind of a consultant and deposit themselves on paper, as opposed to simply aggregated tool results. We DO rely heavily on tools written by our folks and others, but for the usual reasons (false positives, duplication of results for identical issues, inability to correlate issues with business risk) we do not necessarily treat the outputs as gospel. They are part of the overall bag of tools we used to assemble the defect list for each engagement.

Your point about component (non)interchangeability is well taken. We did not attempt to control for use of components. For this variable in particular, the sample size (n=45) was still too small to provide meaningful trend data.

Do you need help?

In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software: Learn more about Pantek's Technical Support offerings Place an order for 24/7/365 Technical Support here Learn more about Pantek, who we are, and what we do Chat with a Live Operator Related securityfocus.com Links bugtraq certification focus-bsd focus-ids focus-ih focus-linux focus-ms focus-sun focus-unix-other focus-virus forensics incidents libnet linux-secnews ms-secnews pen-test secpapers secprog sectools secureshell security-basics securityjobs sf-news vuln-dev webappsec Request more information about Pantek

>
> Overall, I think the paper is a good start, but needs more substiation

Workin' on it...

> As for whether or not it reflects the real world, I'd

The goal of the paper was to begin to frame the app security problem with hard numbers. Are they the *right* numbers? It is too early to tell. But even at this stage, it seems pretty clear that some applications are more secure than others. Hopefully the paper will help decision-makers get away from the traditional black or white choice (I am secure/I am hosed) to one that contains more shades of grey (I ought to focus on areas a-b-c).

To my knowledge, @stake may be the first company to do a serious quantitative study of application security. That doesn't make us the experts, just the first to take a punt at it. :) It is, as you say, a start.

Alex, thanks for the fine critique. You spotted all of the important caveats. :)

Regards,

Andrew

Do you need more help?

In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software: Learn more about Pantek's Technical Support offerings Place an order for 24/7/365 Technical Support here Learn more about Pantek, who we are, and what we do Chat with a Live Operator Related securityfocus.com Links bugtraq certification focus-bsd focus-ids focus-ih focus-linux focus-ms focus-sun focus-unix-other focus-virus forensics incidents libnet linux-secnews ms-secnews pen-test secpapers secprog sectools secureshell security-basics securityjobs sf-news vuln-dev webappsec Request more information about Pantek

PS If there are other folks working in the risk analytics arena who would like to compare notes, send me an off-line reply. I'd be curious to get your perspectives.

PPS Steve, how about a short paper on aggregated stats from Mitre's CVE database? Now THAT would be interesting reading. You'd have to do some digging, I would imagine... Received on Mon Dec 2 18:54:40 2002