Extrapolating the US penetration testing market size

One of the questions I have had a bit following on from my analysis of the Australian penetration testing market, is the implied size of the global penetration testing market.  Or at least, the size of the US penetration testing market, on the assumption that it is going to be the largest.  With a few minutes to spare, I thought I would try to kludge together a number that at least seems plausible given the (admittedly very few) external reference points available.

IBIS World released a research report in August 2012 (the "IT Security Consulting in the US Market Research Report") which provides a couple of free snippets of data - a revenue figure of $5 Billion, and, interestingly, the statement that "there are no companies with a dominant market share in this industry" - which is exactly the conclusion I came to when looking at the Australian penetration testing market.

So there's our first data point:  The US IT Security Consulting Market (2012) is estimated at $5 Billion.  

Global Industry Analysts, Inc have estimated the 2013 global information security products & services market at $104 Billion, and RNCOS has estimated the global IT security market at $96 Billion (both figures from this interesting analysis of the Turkish IT security market).  Not wildly dissimilar numbers which is always a nice start.  A PricewaterhouseCoopers report in 2011 apparently put the estimated market size at $60 Billion, so a bit smaller, but with forecast growth, probably closer to a $75 Billion estimate by 2013.  Gartner has put the global market at $55 Billion in 2011 with a forecast growth path that would imply something like $67 Billion for 2013. 

The US is estimated to make up close to half of all cyber-security spending globally.  Which seems quite plausible when one considers the size of both defence-led Government cyber-security expenditure, and also the size of the economy.  That would put the US cyber-security market into the vicinity of $35-45 Billion for 2013.

One potentially useful stat we can gather from the above, is that IT security consulting, is ~10-15% of the overall IT security market size.

So how do Australia's numbers compare?

This fairly old data set from 2009 has Gartner estimating the Australian IT security market size being about $250 Million.  Let's add on 20%-year-on-year growth since then, and we're at $500 Million-ish today.  Given my previous analysis of the Australian penetration testing market put it at $200-300 Million on its own, I think this is a pretty low estimate.  A 2008 estimate by IDC forecast the market would hit $1.5 Billion by 2011, which actually sounds a bit more workable.

If this is correct, and if my previous penetration testing market estimates are plausible, then at a macro level, organisations are spending 10-20% of their security budget on penetration testing and vulnerability assessment.  This feels a bit high (probably reflecting the fact that less is being spent than the bottom-up estimate of penetration testing expenditure would suggest), and also seems not to match with the US estimate of 10-15% of IT security spend going to consulting.  Given this would contain a great deal of 'non-penetration testing' consulting services, for penetration testing alone, let's go with something closer to 5% to be a bit more conservative.

So as rubbery as these data sets may be, they would suggest that the US penetration testing market is in the $1.5 - 3 Billion range... Which makes it 8-10 times the size of the Australian market, which given the size of the US economy (GDP $15.094 Trillion) is a larger order of magnitude than that, larger than the Australian economy (GDP $1.37 Trillion), would seem to make sense.

And just to recap my favourite point once again... "there are no companies with a dominant market share in the [IT security consulting] industry".  As I said at the end of the Australian analysis, this is a great market to be a part of; and on a global scale that is no different.

Crowdsourcing & the Prisoner's Dilemma

One of the common questions that gets raised in the crowdsourced testing process (eg Bugcrowd) is how it's possible to manage the risk of a tester identifying vulnerabilities, then disclosing them or selling them or using them, outside the parameters of the officially sanctioned test.

While it is presenting an alternative to penetration testing in many cases, it is somewhat more useful to consider the model in the context of the bug bounty programs run by companies like Google.  

The reason for the distinction is that bug bounty programs are aimed at achieving two related, but distinct, goals:

1. To have vulnerabilities that would have been identified anyway (ie through unauthorised testing, or through incidental testing for a third party), be responsibly disclosed; and
2. To have additional vulnerabilities identified by encouraging additional testing, and corresponding responsible disclosure.

That first group is often not considered as a goal of a penetration test - the likelihood that any system of interest is constantly being subject to security analysis by Internet-based users with varying shades of grey- or black- hats, seems to often to be overlooked.  

With the risk of stating the obvious, the reality is that every vulnerability in a given system, is already in that system.  Identifying vulnerabilities in a system does not create those weaknesses - but it is true that it may increase the risk associated with that vulnerability as it transitions from being 'unknown' to being 'known' - depending on who knows it.  

Feel free to spread this video around.

To use Donald Rumsfeld's categorisation, we could consider the three groups as follows:

1. Known Knowns: Vulnerabilities we know exist and are known in the outside world (publicly disclosed or identified through compromise);
2. Known Unknowns: Vulnerabilities that we know exist, and are unsure if they are known in the outside world (either identified by us; or privately disclosed to us);
3. Unknown Unknowns: Vulnerabilities that we don't know exist, and are unsure if they are known in the outside world (which is the state of most systems, most of the time).

What crowdsourcing seeks to do, is to reduce the size of the 'unknown unknown' vulnerability population, by moving more of them to the 'known unknown' population so that companies can manage them.  The threat of a 'known unknown' is significantly lower than the threat of an 'unknown unknown'.

Which brings us to the risk that a vulnerability identified through a crowdsourced test, is not reported, and hence remains an 'unknown unknown' to us.  The risk of non-disclosure of vulnerabilities identified through a crowdsourced test is effectively mitigated by game theory - it is somewhat similar to the classic 'Prisoner's Dilemma': 

The Prisoner's Dilemma is a classic of game theory, demonstrating why individuals may not cooperate, even if it is in their best interests to do so.  The Dilemma goes like this:

"Two members of a criminal gang are arrested and imprisoned. Each prisoner is in solitary confinement with no means of speaking to or exchanging messages with the other. The police admit they don't have enough evidence to convict the pair on the principal charge. They plan to sentence both to a year in prison on a lesser charge. Simultaneously, the police offer each prisoner a Faustian bargain. If he testifies against his partner, he will go free while the partner will get three years in prison on the main charge. Oh, yes, there is a catch ... If both prisoners testify against each other, both will be sentenced to two years in jail."

Effectively, the options are as presented in this table:

The beauty of the dilemma, is that as they cannot communicate, each prisoner must evaluate their own actions without knowing the actions of the other.  And for each prisoner, they get a better outcome by betraying the other prisoner.  For Prisoner A looking at his options, if Prisoner B keeps quiet, Prisoner A has the choice of 1 year in jail (if he also keeps quiet) or no jail time at all (if he testifies against Prisoner B). Hence, testifying gives a better outcome.  And if Prisoner B testifies against him, Prisoner A has the choice of 3 years in jail (if he keeps quiet) or 2 years in jail (if he also testifies)... again, testifying gives a better outcome.

Hence, economically rational prisoners will not cooperate, and both prisoners will serve 2 years in prison, despite that appearing to be a sub-optimal outcome.

What does this have to do with crowdsourcing?

In crowdsourcing there are obviously far more than 2 participants; but the decision table we are interested in, is as it is relevant to any particular tester.  The situation they face is this:

Essentially, each tester only knows the vulnerabilities they have identified.  They do not know who else is testing, or what those other testers have discovered.

Only the first tester to report a vulnerability gets rewarded.

Any tester seeking to 'hold' an identified vulnerability for future sale/exploitation (as opposed to payment via the bounty system) has to be confident that the vulnerability was not identified by anyone else during the test, since otherwise they are likely to end up with nothing - the vulnerability gets patched, plus they don't get any reward.  

Since Bugcrowd tests to date have had large numbers of participants, and have found that over 95% of vulnerabilities are reported by more than one tester, this is a risk that will rarely pay off.

As a result, economically rational testers will disclose the vulnerabilities they find, as quickly as possible.  

For organisations getting tested, cliched as it is, the crowd truly does provide safety in numbers.

​Disclaimer: I'm an Advisor to Bugcrowd.  

Being an Entrepreneur: The Best Advice I Received

Back in the Spring of 2000, I was finishing University and figuring out what to do next.  Two main paths lay before me: Follow many of my peers into the strategy consulting world (McKinsey, BCG, Booz Allen & Hamilton, Bain, AT Kearney and others), or venture out into the great unknown of starting a business.  I interviewed with the majority of the consulting firms, and received a couple of offers (although not the ones I really wanted).  Weighing up my options, it seemed that it was as good a time as any to try my hand at starting a business, and in the worst case scenario, I'd go back to that consulting pool in a year or two's time.

I started the company which became "SIFT" with a good friend from University, who had similar aspirations to set our own agenda and prove that we could do it.  We were very fortunate with SIFT that almost every early recruitment decision we made, worked out better than we could possibly have imagined, with that core team sticking with the company for many years, allowing us to build the company in their image, and believing passionately about the importance of doing things well - a passion that we could never have intended to 'teach', but that arose from a group of like minded people, seeking to make their mark on the world.

We grew slowly at first, then picked up pace in about 2005 once we had been in the market for a while, and started to be quite well known.  By this point I was already a regular presenting on the conference circuit, and for five years had attended virtually every industry event I could get into without paying.  Conferences, product launches, vendor drinks, breakfast forums, I was there, telling people who we were and what we did.

In 2007, the opportunity presented itself to complete our first corporate transaction, the acquisition of Safecoms Secure IT.  As with so much of the business, this arose primarily through the relationship between the founder of Safecoms Secure IT, and myself.  I have always believed that the market in Australia is too small to treat 'competitors' badly.  Your competitor today could easily be your client or partner tomorrow.  

We acquired Safecoms Secure IT, some staff and some clients, and set about making 2007 the strongest year we had yet recorded.

At the end of 2008, I met the Stratsec founders in Melbourne, and immediately felt I was with friends.  Within hours of meeting we had started putting together the framework for how SIFT and Stratsec could come together to form the largest information security consulting firm in Australia; and in May 2009, that's exactly what happened.  Our 'exit strategy' at that point - loose as it was - was that we expected within a 3 year timeframe, to be at a point where we could look at being acquired.  Within 18 months, we were sold.

After approaches by a number of potential acquirers in early 2010, we decided that a piecemeal approach responding to our suitors was both inefficient and likely to result in a sale on someone else's terms rather than our own.  I put together our Information Memorandum, and we took it both to companies who had already expressed interest, as well as others who we felt would be a good fit.  

BAE Systems Australia was the last company into the process; but ultimately were the successful acquirer.  The strategic fit was unquestionable, and their integrity, approach, and growth ambitions, were all a fit for how we did business.  When Stratsec was acquired at the end of 2010, we had about 60 staff.  By December 2012, that had doubled to about 120 and was continuing to grow rapidly, providing some great opportunities for our young leadership team and creating a genuine 'Tier 1' information security firm in the Australian market.

In December 2012, I finished at BAE Systems Detica (as we had rebranded Stratsec in the weeks before my departure), bringing to a close an adventure which ran for 12 years, 3 months, 2 weeks and a few days.  It is something I am very proud of, to have started, acquired, merged, and ultimately sold the company successfully.  Very few entrepreneurs get such a 'neat' end-to-end start/build/sell experience.   

As time closed in on my departure date, a colleague in the industry, upon hearing that I was moving on, said “That’s a shame, Nick… you were always one of the guys who gave a shit.”  It was meant sincerely, and I took it that way.  And as I explained at my farewell party, I am more than happy for that to be my main contribution to the culture of Stratsec.  These aren’t just projects, it is real life.  At Stratsec, that meant that testing a system to ensure a malicious hacker couldn't break into your bank account or your health records, was more than just a 'testing project' - after all, it could just as easily be my, or your, account or record that could be compromised.  And now at Delling Advisory, I see the same thing, when I look at how enmeshed many founders are with the businesses they have started and run.  These are not just companies to be sold; they are years, if not decades, of long days and nights, and trying to make the business  a little bit better every day.

A number of people have asked me what my greatest 'lessons learnt' were, or greatest inspirations, to help make their entrepreneurial journey shorter and easier.  The best advice I received was from Bob Metcalfe (inventor of Ethernet and founder of 3Com).  Of course, the advice wasn't directly from Bob to me, it was in the form of his 1999 article "Invention is a Flower, Innovation is a Weed" (still online at http://www.technologyreview.com/featuredstory/400489/invention-is-a-flower-innovation-is-a-weed/). 

A snippet from the article:

I have a six-story townhouse in Boston overlooking MIT on the Charles River. I often invite young engineers and would-be entrepreneurs over to schmooze. Many of them tell me my townhouse is beautiful and they hope to invent something like Ethernet that will get them such a house.

The picture they have in their heads is of me lounging around on the beanbag chairs in a conference room at Xerox PARC in 1973. They see me having this idea for a computer network and submitting it as an invention proposal to Xerox. Then they envision me putting my feet up and letting the royalties roll in until I have enough to come up with the down payment on the townhouse with the river view.

My picture - the actual picture - is different. It’s the picture of innovation rather than invention, the weed instead of the flower. In my picture it’s the dead of winter and I am in the dark in a Ramada Inn in Schenectady, New York. A telephone is ringing with my wake-up call at 6 a.m., which is 3 a.m. in California, where I flew in from last night. I don’t know yet where I am, or where that damn ringing is coming from, but within the hour I’ll be in front of hostile strangers selling them on me, my company, and its strange products, which they have no idea they need.

If I persist, selling like this for 10 years, and I do it better and better each time, and I build a team to do everything else better and better each time, then I get the townhouse. Not because of any flowery flash of genius in some academic hothouse.

And that is exactly my experience of building Australia's largest information security consulting firm from the ground up.  That's Bob's lesson number 1 - "Selling Matters."

"Don't Hire - Recruit", and "Be an Entrepreneur, Not a Visionary" are two others that have resonated with my journey; but they're all great lessons.  I cannot recommend highly enough that you read it, think about it, and put it into action.