Having recently spoken with a number of large corporations, both in the US and Japan, I have found there is definitely a shortage of engineers in the area of Hadoop and Data/Research Scientists. With the growing interest in the transformative technology called Hadoop and the topic of Big Data applications, many enterprises are scrambling to find developers who understand how to deploy, support and ultimately take advantage of the opportunity this technology represents. Once the Hadoop platform is deployed, it takes high level mathematicians, or data scientists, to write the algorithms that will make effective use of the vast amount of information within the repository, resulting in real business value. It will take some time for the talent supply in these two critical areas to catch up with demand before the opportunity this technology represents can be utilized by the vast majority of corporations. Because of this we predict financial services, telecommunications, US Government and large retailers taking the lead in harnessing Big Data for their respective use cases.
While the number of computer engineers struggles to catch up, we are seeing entrepreneurial startup companies, like Lucid Imagination, begin to offer Big Data development platforms that abstract some of the complexity of the underlining technology – making it easier to deploy. These tools will take some maturing but they promise to provide corporations of all sizes the ability to enjoy the benefits associated with this new wave of information-based applications.