How much and where does Crowd Size really matter?

This is part 1 of our series on Crowd Metrics & Crowd Quality

The size of a crowd nowadays has become a major marketing argument for most crowdsourcing and specifically crowd testing vendors, and a strong emphasis is put on such numbers once these vendors are featured in research firms' reports and media articles. As a logical consequence, the size of the crowd is mostly one of the first questions asked and used as a major selection criteria by most enterprises today when it comes to vendor selection. If you search the Internet for 'crowd size' you'll find a lot about the highly disputable facts on the size of the crowd attending Donald Trump’s inauguration, but drilling further down targeting publications related to crowdsourcing there is very little to discover. How come?

Possibly this is a topic that hasn't been researched well yet, which vendors so far preferred not to dive into too much and therefore is still rather blind-spotted. With crowd sizes in crowdsourcing, like with 'The Donald's' inauguration crowd, it is not really transparent 'who counts what'. So I thought it might be worth sharing some of our more than 5 years experience we made in the crowd testing and more recently in other crowdsourcing areas of the digital assurance spectrum of activities.

First of all, the total size of a crowd any kind of crowdsourcing services vendor holds is relevant only to provide some kind of indication about the overall crowd scale. However it doesn't have much to do with crucial matters such as the ability to identify, select and engage the desired number of project participants matching the required criteria which could be geographic, demographic, skills and background; device/OS configurations owned, mobile network subscribed to; being a customer of a given company, product or service etc.

Imagine a vendor claiming to have a crowd of 6-digit-numbers members and a major percentage of them are sitting in one country (mainly India?) only, or another vendor claiming a crowd size in large numbers as well but his member database does not provide sufficient profile details. Have you also considered the possibility that a large portion of registered members of a given crowdsourcing platform might have actually been inactive on the platform since several months or even years ago?

What is the value of a crowd of which 90% has never been utilized, how many of its members might not be dormant but rather not reachable any longer through the email account they provided? What about the qualification and performance ratings of crowd members? These are only a few example questions which highlight the complexity of the crowd-matter which is obviously approached in a too much simplified way when looking only at the total crowd size as an indicator of a vendor's capabilities.

Some companies really believe that a vendor's overall crowd size matters a lot when they need to recruit only 20 crowd testers for a functional test of a mobile app, and this is just one of many examples we have seen in the past where customers based their decisions on invalid assumptions.

This series of blog posts is here to help understand how to assess the potential of a given crowd for different types of engagement.

In the next posts which will follow shortly, I will introduce the term 'crowd quality' with a set of truly relevant metrics you need to look at.


Authored by:

Dieter Speidel, CEO


Write new comment

Comments (0)

No comments found!