A Market Size Formula for the Security Assurance of Everything

Last year, I wrote an article entitled "Penetration testing market analysis: Where is all the revenue?" looking at the Australian penetration testing market, and estimating the size of the market at $200 - 300 Million.  Since then I've had any number of requests to extend the analysis, to answer the question of how big the US penetration testing market is; of how big the global penetration testing market is; of how big the global security assurance market is; and various other slicings and dicings of the data.

One of the more interesting questions that came up was whether a high-level percentage figure could be identified to then provide a sense of the security-relevant market potential of new areas of development.  I thought I'd play with some numbers and see what popped out, and decided to share it here to solicit thoughts from anyone else interested in the topic.  It's important to note that this is definitely not an economically or statistically rigorous approach to the topic; but rather is taking a handful of numbers and smashing them together to see what pops out, and to see if perhaps there's a meaningful trend that we can interpret.  

Whereas my analysis of the penetration testing market was bottom up, when we're looking at something the size of the global IT market, it's simply too big to do that way, so we need to start from the top and work down.

At the top, we have some really big numbers.  Gartner has estimated that the worldwide spend on enterprise IT was $2,700 Billion in 2012.  (Note that I'm not entirely sure whether this figure includes or excludes personnel costs, which could throw a spanner in the works for the rest of this piece... but just go with it for now).

In terms of how security fits into these big numbers, Gartner has also provided some analysis on that, suggesting that in 2010, approximately 5% of the enterprise IT budget was spent on security, with that 5% breaking down further to:

• Personnel (37%)
• Software (25%)
• Hardware (20%)
• Outsourcing (10%)
• Consulting (9%)

(And yes, that adds up to 101%; obviously that's the result of rounding; we can live with it)

If 5% of the $2,700 Billion market was spent on security, that would give us about a $135 Billion IT security market.  At first glance this looks high, since most estimates have the security market at between $75-100 Billion, but one notable difference is the inclusion of internal staff - "personnel" - in that figure.  Adjusting to take that out, the market size becomes $98.5 Billion, which is spot on with most other estimates.  Could just be luck, but it's a nice starting point to at least have a little internal consistency in our numbers.

Taking it one step further, the consulting figure (9%) would imply a consulting market size of $12.15 Billion, which seems pretty reasonable (for comparative data points, IBIS World has the Australian IT Security Consulting market at $2 Billion, and the US IT Security Consulting market at $5 Billion, so to add another $5 Billion for 'rest of world' seems about right, and hey presto... same number).

Security assurance activities are mostly going to fall within the 'consulting' bucket, but there will likely be some cross-over.  Some companies have internal penetration testing teams (so it would fall into 'personnel'), others will spend big on automated scanners and the like (so it would fall into 'software' and 'hardware') and others may categorise it as some kind of managed service (which would put it into 'outsourcing').  

My estimate would be that at least 30% of IT Security Consulting spend goes towards security assurance activities.  So that's 30% assurance of 9% consulting of 5% of enterprise budget of $2,700 Billion, and crunching that all together gives us a figure of about $3.65 Billion as the security assurance consulting global market.  (I think it would be higher if associated products like vulnerability assessment tools/scanners and the like were included).  The good news is that this figure is also pretty well aligned with other research out there about market sizing.

It's also notable that the budget figure we're using - 5% - is from 2010.  More recent estimates have thrown out numbers that range from a little higher (5.6% - FT.com 2011) to a lot higher (7.5% - Wisegate 2013).  The 7.5% figure would make the security assurance consulting global market closer to $5.5 Billion, which is plausible, but probably stretching it a bit so I'm inclined to be conservative and stick with 5% as a macro-level average... but certainly note that as budgets rise, there is a substantial impact on the security market.

To go from the macro- global level, to a country level, let's look at Australia and see what happens.  Borrowing a data set from SMS Management & Technology's market presentation, their merger of data from Gartner, Forrester, and their own analysis, puts the Australian enterprise IT market size at $47.1 Billion.  Note that this excludes personnel costs.  5% on IT security would mean $2.355 Billion.  Adjusting for the personnel missing, and then just taking out the consulting chunk, we would have an IT security consulting market of about $340 Million, and a security assurance consulting market of about $100 Million.  That's lower than I've previously estimated from a bottom-up view (my estimate is $200-300 Million) but I think in part that's a function of the fact that we actually have a pretty substantial and dynamic security assurance market in Australia, which could easily account for 50% of the consulting spend rather than 30% that I've apportioned globally.  

The final leap of faith here is to look at the ultimate 'trickle down' from a macro level market size, to the security assurance consulting market we are operating in.  And the magical number appears to be something in the range 0.14% - 0.20% of a macro level IT market size, trickles down to security assurance.

So, and here comes the magic, that would mean we have a handful of prospective security assurance consulting markets out there including:

Mobile Ecosystem - Tablet market ~$35 Billion, Smartphone market ~$150 Billion, Application market $25 Billion - implied security assurance consulting market $300 - 400 Million.  

Internet of Things - IDC estimated market size $4.8 Trillion (Read that number again.  That's Trillion.) - implied security assurance consulting market between $6 - 10 Billion.

I'll leave it there.  Partly because I have to go and start an Internet of Things security company right now.

Extrapolating the US penetration testing market size

One of the questions I have had a bit following on from my analysis of the Australian penetration testing market, is the implied size of the global penetration testing market.  Or at least, the size of the US penetration testing market, on the assumption that it is going to be the largest.  With a few minutes to spare, I thought I would try to kludge together a number that at least seems plausible given the (admittedly very few) external reference points available.

IBIS World released a research report in August 2012 (the "IT Security Consulting in the US Market Research Report") which provides a couple of free snippets of data - a revenue figure of $5 Billion, and, interestingly, the statement that "there are no companies with a dominant market share in this industry" - which is exactly the conclusion I came to when looking at the Australian penetration testing market.

So there's our first data point:  The US IT Security Consulting Market (2012) is estimated at $5 Billion.  

Global Industry Analysts, Inc have estimated the 2013 global information security products & services market at $104 Billion, and RNCOS has estimated the global IT security market at $96 Billion (both figures from this interesting analysis of the Turkish IT security market).  Not wildly dissimilar numbers which is always a nice start.  A PricewaterhouseCoopers report in 2011 apparently put the estimated market size at $60 Billion, so a bit smaller, but with forecast growth, probably closer to a $75 Billion estimate by 2013.  Gartner has put the global market at $55 Billion in 2011 with a forecast growth path that would imply something like $67 Billion for 2013. 

The US is estimated to make up close to half of all cyber-security spending globally.  Which seems quite plausible when one considers the size of both defence-led Government cyber-security expenditure, and also the size of the economy.  That would put the US cyber-security market into the vicinity of $35-45 Billion for 2013.

One potentially useful stat we can gather from the above, is that IT security consulting, is ~10-15% of the overall IT security market size.

So how do Australia's numbers compare?

This fairly old data set from 2009 has Gartner estimating the Australian IT security market size being about $250 Million.  Let's add on 20%-year-on-year growth since then, and we're at $500 Million-ish today.  Given my previous analysis of the Australian penetration testing market put it at $200-300 Million on its own, I think this is a pretty low estimate.  A 2008 estimate by IDC forecast the market would hit $1.5 Billion by 2011, which actually sounds a bit more workable.

If this is correct, and if my previous penetration testing market estimates are plausible, then at a macro level, organisations are spending 10-20% of their security budget on penetration testing and vulnerability assessment.  This feels a bit high (probably reflecting the fact that less is being spent than the bottom-up estimate of penetration testing expenditure would suggest), and also seems not to match with the US estimate of 10-15% of IT security spend going to consulting.  Given this would contain a great deal of 'non-penetration testing' consulting services, for penetration testing alone, let's go with something closer to 5% to be a bit more conservative.

So as rubbery as these data sets may be, they would suggest that the US penetration testing market is in the $1.5 - 3 Billion range... Which makes it 8-10 times the size of the Australian market, which given the size of the US economy (GDP $15.094 Trillion) is a larger order of magnitude than that, larger than the Australian economy (GDP $1.37 Trillion), would seem to make sense.

And just to recap my favourite point once again... "there are no companies with a dominant market share in the [IT security consulting] industry".  As I said at the end of the Australian analysis, this is a great market to be a part of; and on a global scale that is no different.

Crowdsourcing & the Prisoner's Dilemma

One of the common questions that gets raised in the crowdsourced testing process (eg Bugcrowd) is how it's possible to manage the risk of a tester identifying vulnerabilities, then disclosing them or selling them or using them, outside the parameters of the officially sanctioned test.

While it is presenting an alternative to penetration testing in many cases, it is somewhat more useful to consider the model in the context of the bug bounty programs run by companies like Google.  

The reason for the distinction is that bug bounty programs are aimed at achieving two related, but distinct, goals:

1. To have vulnerabilities that would have been identified anyway (ie through unauthorised testing, or through incidental testing for a third party), be responsibly disclosed; and
2. To have additional vulnerabilities identified by encouraging additional testing, and corresponding responsible disclosure.

That first group is often not considered as a goal of a penetration test - the likelihood that any system of interest is constantly being subject to security analysis by Internet-based users with varying shades of grey- or black- hats, seems to often to be overlooked.  

With the risk of stating the obvious, the reality is that every vulnerability in a given system, is already in that system.  Identifying vulnerabilities in a system does not create those weaknesses - but it is true that it may increase the risk associated with that vulnerability as it transitions from being 'unknown' to being 'known' - depending on who knows it.  

Feel free to spread this video around.

To use Donald Rumsfeld's categorisation, we could consider the three groups as follows:

1. Known Knowns: Vulnerabilities we know exist and are known in the outside world (publicly disclosed or identified through compromise);
2. Known Unknowns: Vulnerabilities that we know exist, and are unsure if they are known in the outside world (either identified by us; or privately disclosed to us);
3. Unknown Unknowns: Vulnerabilities that we don't know exist, and are unsure if they are known in the outside world (which is the state of most systems, most of the time).

What crowdsourcing seeks to do, is to reduce the size of the 'unknown unknown' vulnerability population, by moving more of them to the 'known unknown' population so that companies can manage them.  The threat of a 'known unknown' is significantly lower than the threat of an 'unknown unknown'.

Which brings us to the risk that a vulnerability identified through a crowdsourced test, is not reported, and hence remains an 'unknown unknown' to us.  The risk of non-disclosure of vulnerabilities identified through a crowdsourced test is effectively mitigated by game theory - it is somewhat similar to the classic 'Prisoner's Dilemma': 

The Prisoner's Dilemma is a classic of game theory, demonstrating why individuals may not cooperate, even if it is in their best interests to do so.  The Dilemma goes like this:

"Two members of a criminal gang are arrested and imprisoned. Each prisoner is in solitary confinement with no means of speaking to or exchanging messages with the other. The police admit they don't have enough evidence to convict the pair on the principal charge. They plan to sentence both to a year in prison on a lesser charge. Simultaneously, the police offer each prisoner a Faustian bargain. If he testifies against his partner, he will go free while the partner will get three years in prison on the main charge. Oh, yes, there is a catch ... If both prisoners testify against each other, both will be sentenced to two years in jail."

Effectively, the options are as presented in this table:

The beauty of the dilemma, is that as they cannot communicate, each prisoner must evaluate their own actions without knowing the actions of the other.  And for each prisoner, they get a better outcome by betraying the other prisoner.  For Prisoner A looking at his options, if Prisoner B keeps quiet, Prisoner A has the choice of 1 year in jail (if he also keeps quiet) or no jail time at all (if he testifies against Prisoner B). Hence, testifying gives a better outcome.  And if Prisoner B testifies against him, Prisoner A has the choice of 3 years in jail (if he keeps quiet) or 2 years in jail (if he also testifies)... again, testifying gives a better outcome.

Hence, economically rational prisoners will not cooperate, and both prisoners will serve 2 years in prison, despite that appearing to be a sub-optimal outcome.

What does this have to do with crowdsourcing?

In crowdsourcing there are obviously far more than 2 participants; but the decision table we are interested in, is as it is relevant to any particular tester.  The situation they face is this:

Essentially, each tester only knows the vulnerabilities they have identified.  They do not know who else is testing, or what those other testers have discovered.

Only the first tester to report a vulnerability gets rewarded.

Any tester seeking to 'hold' an identified vulnerability for future sale/exploitation (as opposed to payment via the bounty system) has to be confident that the vulnerability was not identified by anyone else during the test, since otherwise they are likely to end up with nothing - the vulnerability gets patched, plus they don't get any reward.  

Since Bugcrowd tests to date have had large numbers of participants, and have found that over 95% of vulnerabilities are reported by more than one tester, this is a risk that will rarely pay off.

As a result, economically rational testers will disclose the vulnerabilities they find, as quickly as possible.  

For organisations getting tested, cliched as it is, the crowd truly does provide safety in numbers.

​Disclaimer: I'm an Advisor to Bugcrowd.  

Penetration testing market analysis: where is all the revenue?

I was recently sitting at the Australian Technology Park having a cup of coffee with Casey Ellis, co-founder of Bugcrowd, chatting about upcoming investor presentations.  We worked our way on to market sizing, and found that we had both had the same experience when attempting to do a 'bottom up' sizing of the penetration testing market in Australia.  The problem that we both came across, was that even using fairly conservative numbers as to the amount companies are spending on penetration testing, the amount of theoretical penetration testing revenue sloshing about in the market simply does not align with the revenue of the service providers in this space, or simply with the number of testers providing these services.

[Incidentally, I had brief flashbacks to my case-study interviews with strategy consulting firms before I started SIFT... where I had awesome questions like: 

• "Estimate the size of the market for salmon in the United Kingdom"; and
• "Estimate the number of PCs imported to Australia each year".]

Back to the penetration testing market... 

Let's start with the big guys.

ASX 20

Of the ASX20, which includes companies in financial services, materials/mining, energy, consumer staples, telecommunications and healthcare, my back-of-the-envelope estimates would suggest that the biggest spenders would spend about $4 million annually on penetration testing, and the lowest spenders would spend about $100K annually.  Putting together the expenditure of the whole group, I estimate it works out at pretty close to a neat $20 million across the 20 companies.

And of course, the ASX20 is - as its name suggests - just the 20 largest companies by market capitalisation on the ASX.  There are a total of 2,157 companies listed on the ASX (when I downloaded the list a moment ago), all of whom you could argue have some degree of obligation to their shareholders to ensure the security of their data and systems, with penetration testing being a pretty common response to that obligation.  For argument's sake, lets say less than half of them do anything, so 1,000 companies.  And let's assume that averaged across that many organisations, the average spend on penetration testing is $50K per annum.  That's another $50 million into the annual penetration testing market.

Let's look at some other big-spending sectors where some reasonably neat figures are available (about the size of the sector; if not the amount spent):

Financial Services

I'd estimate that about 60-70% of the ASX20 spend is coming from the financial services companies in the group who were some of earliest adopters of penetration testing as a service, and continue to be the 'anchor tenant' for the industry.

According to APRA, at the end of 2012, there were 19 Australian banks, 8 foreign subsidiary banks, and 40 branches of foreign banks.  On top of these, there were 91 credit unions and 9 building societies.  There are also a handful of 'miscellaneous' companies like payments clearing, 'specialist credit card institutions' and 'purchased payments facilities' who are also significant market participants.

So that's an extra 170-ish financial services companies who are probably getting penetration testing completed to a greater or lesser extent.  Even if we rule out the 'branches of foreign banks' (as many of them will have their penetration testing managed by the global head office and hence delivered from overseas), we've still got about 130.  Chop out the group already counted in the ASX20, and we've got about 125.  Now let's be super-conservative and say that they will spend only 10% of the amount that the larger companies will spend; or a meager $100K per institution.  That's another $12.5 million into the annual penetration testing market.

Take a moment to consider that according to the Australian Bureau of Statistics, at the end of the 2010-11 year, there were over 164,000 businesses in Australia classified as 'financial and insurance services'.  In the calculations above we covered about 200 of them; admittedly the biggest, but it still leaves a vast number who have data to protect, and some of whom certainly have some penetration testing done.  (If just 2% of them spend just $5K each, that's another $15 million into the budget).

Government

Federal, State and even Local Government are covered by a range of policies explicitly requiring independent penetration testing.  One of the most succinct is that of the Victorian Government - SEC STD: Penetration testing which states that:

According to vic.gov.au's Contacts & Services directory, there are 521 distinct entities within the Victorian Government, for which 259 unique URLs are provided.  For example, the letter 'A'...  

As per policy, each of these needs at least annual independent penetration testing.  Let's use our average across the set (covering both infrastructure and applications) of just $20K per annum.  That gives us about another $6 million for our penetration testing budget.

To avoid the pain of digging out the numbers for all the other states and territories, let's make a broad assumption that all the other state and territory governments added together, sum to three times the size of Victoria's, in terms of Internet-facing infrastructure (which given it include NSW & QLD, plus the rest, seems reasonable).  Let's also assume that they have a similar intent to test everything annually.  So that's another $18 million to the budget.  That number feels high, so let's include all local government, councils etc across the country as well in that figure.

And of course there is also Federal Government.  It's possible to download a list of all registered contracts with keywords like 'penetration testing' or 'security testing' at https://www.tenders.gov.au/?event=public.CN.search, but these lists are woefully incomplete when trying to get a picture of the size of the market.  The Federal Government side of things is also somewhat obscured by the fact that at least some of the vulnerability assessment and penetration testing completed is performed by the Defence Signals Directorate (DSD).  Rather than tie myself in knots trying to work it through, I'll take a short-cut and assume it's the same as Victoria: $6 million annually, across all government agencies including the Defence Department.

E-Commerce / Payments

The Payment Card Industry Data Security Standard (PCI DSS) requires penetration to be completed at least annually for in-scope systems and organisations. 

There are approximately 200,000 websites in the .au domain space with 'shopping cart' functions.  Mmany of those will be using PCI compliant externally-hosted shopping carts so probably don't get penetration testing completed themselves.  But let's say just 10% of e-commerce websites with 'shopping cart' functions get penetration tested each year.  That's 20,000 websites.  Most of these are probably pretty small, so let's say they are just $10K penetration tests.  That's another $20 million in the budget.

We'll assume that the vast number of companies covered by PCI DSS, but who don't have a distinct 'shopping cart' function so aren't included in the figures above, are covered elsewhere in one of the figures we've already looked at.

Education 

There are 44 universities in Australia, and another half-a-dozen miscellaneous self-accrediting higher education institutions (ie theological colleges, maritime college etc), giving us a nice neat 50.

There are then at least another 100 state and territory accredited educational organisations, plus TAFEs and the like.  There are thousands of schools.

Given universities'... errr... 'creative' student population, they have a bigger need than most of the others here.  Let's assume $100K per annum for the universities, which is $5 million in total to the budget.

For the thousands of schools, TAFEs, and other miscellaneous bodies, it's hard to know where to start, so let's just allocate the entire sector $25 million and be done with it.  If there are 5,000 schools across the country that's only $5K of testing per school, so pretty conservative, although I'm cognisant of the fact that far-flung country-shed classrooms are unlikely to be having this testing done.

Information & Communications Technology (inc Software)

One of the larger consumers of penetration testing services is the broad and large ICT industry - and in this I also include companies developing software for sale to others, who therefore have a requirement for security assurance of that product prior to taking it to market.  It is also the fourth largest industry sector contributing to Australian GDP and employs 291,000 people in Australia. According to the Australian Bureau of Statistics, at the end of the 2010-11 year, there were 18,854 businesses operating in the Information, Media & Technology classification. 

Let's just say 1% of these companies, spend $100K annually on penetration testing.  That's close enough to another $20 million.

The rest

And we haven't even touched industry sectors like healthcare, resources (in the midst of all the 'China APT' news), legal, accounting, professional services, let alone the hundreds of thousands of small and medium sized businesses in this country, at least some of whom are spending some money on penetration testing.  

Adding it all up

So using this logic, there's a spend of something like $200-300 million on penetration testing, annually, in Australia.  Given the massive slabs of Australian business that are not covered in the figures above, even with the odd wayward assumption or double counting here and there, it seems reasonable.

And this is where the trouble starts.  Where is it going?

Many jurisdictions have bodies similar to the ACCC who are responsible for monitoring the misuse of market power.  In some of these jurisdictions, they have put numbers to what 'substantial market power' means, and a 'minimum' threshhold for considering a company to have an influential market position.  The best figures I could find are from Hong Kong, who discuss using 40% as an indicator of 'substantial market power', and 25% as the 'minimum' threshhold before being particularly interested in a company's market position.  Working with these:

• Taking the 40% figure, we'd be looking for a company with $80-120 million in penetration testing revenue, annually, in Australia.  They don't exist.  No big deal, it just means we don't have a company with 'substantial market power'.
• Taking the 25% figure, we'd be looking for a company with $50-75 million in penetration testing revenue, annually, in Australia.  They still don't exist.  So we don't have any real competition concerns in the market, which is healthy.
• For argument's sake, let's take a 10% figure, so we'd be looking for a company with $20-30 million in penetration testing revenue, annually, in Australia.  I'm still doubtful any service provider in Australia operates at that level.

If I'm right, and there is not a single company in Australia with 10% of the penetration testing market, who is delivering all these penetration tests?  Or is it that the numbers above are fundamentally incorrect because organisations just don't do as much penetration testing as they should (under policy, regulation, best practice etc)?

Let's take another angle on this.  Using $200 million as the market size, and a pretty standard average consulting rate of $1,500/day, there are about 133,333 days worth of consulting-level penetration testing to be delivered each year, which would require about 610 full time penetration testers in service provider organisations.  They aren't there either.

One thing I am confident of is that there is also an extremely long tail when it comes to suppliers of these services.  That is, there is a very large set of companies who each provide a very low portion of the services overall consumed in the market.  A great many miscellaneous ICT service providers (of which as per above there are many thousands) provide security related services such as penetration testing to their existing client base, with varying levels of quality.  Because of the large numbers, if 1,000 of these companies provide $100K of penetration testing services each, that could make up $100 million of the market total.

Another interesting question is how big the market would be if everyone was following 'best practice'.  At present, there is far from anything like consistency when it comes to the amount that organisations are spending on IT security, let alone on a sub-set of the topic such as penetration testing.  Near-identical banks can quite plausibly be spending amounts on penetration testing that are out by a factor of 10.  Where one bank spends $2 million; another spends $200,000.  There are also a great many companies - including those no doubt in lists like the ASX 200 - who simply do not have penetration testing completed at any meaningful level.

If all Government agencies were following policies and had every system tested annually; and all PCI-relevant organisations had penetration testing completed annually; and all ICT companies had their software and hardware tested before releasing it to market... etc, then the figures above could easily double to $500 million plus, annually.

So we have a $200-300 million market (much of which is probably only now coming to market for the first time), with a half-billion dollar opportunity, with no company in a position of market dominance, and an  under-supply of qualified penetration testers to deliver the services.  

Pretty compelling.  Want to buy a penetration testing company?  Call me.