Why is it so difficult to find Data Scientists?

Remember my friend Sue, the self-proclaimed big data "Stuper User"? Well, as you may recall, Sue's company was interested in customer data analytics and the ability to extract insights for their marketing campaigns. The problem is, there is a shortage of people with the necessary skills to provide those insights.

This is not surprising given that data science is only now coming into its own. Harvard Business Review called the Data Scientist the "sexiest job of the 21st century" but not just anyone can be a Data Scientist: as Joel greenhouse writes in the Huffington Post, statistical know-how is the foundation of the profession.

This is what differentiates an analyst from the scientist: the analyst will run the queries and statistical tests, but the scientist will design experiments and tease out those oh-so-valuable insights that everyone is talking about. And this, folks, is why it is so difficult to find an individual who:

1. Understands your business and industry;
2. Has the necessary statistical background;
3. Is technically savvy enough to understand how databases work;
4. Can write the programs used to test hypotheses (such as SAS programming language, R, Erlang, etc.);
5. Is able to craft simple and coherent reports that are actionable.

Educational institutions are falling over themselves to create/capitalize on master's level certificate and graduate degree programs in data science, business intelligence, or business analytics, etc. And they're apparently not cheap! They range from $10,000 to $60,000 and anywhere from 10 months to a couple of years in duration. Time will tell if these programs are graduating data scientists or analysts.

In the meantime, Sue's company will continue to search for someone who can fill its need for customer insight in a market that has a shortage of available candidates.