The Future of Big Data and What that Means for HiringBig data is no longer just a buzzword. It’s a key technology that is penetrating every function across every organization. IBM reports that 2.5 quintillion bytes of data are created every single day; a number that, according to the McKinsey Global Institute, is doubling every three years. But are businesses successfully harnessing the power of all that data? Or is it sitting in siloes, waiting in earnest to be transformed into valuable insight? With a dearth of skilled professionals and an ever-growing demand for implementation, what does the future of big data mean for business, and how does it impact hiring?

The Future of Big Data

Big data became a buzzword in the last several years because of its bold promise to entirely transform the business world. Today, experts agree that this promise was not overblown. Companies worldwide are using big data and analytics to overhaul business processes, revolutionize their marketing strategies, and set the stage for other trends such as artificial intelligence, deep learning, machine learning, and the Internet of Things.

One of the most prominent reasons why big data has taken off so quickly is the widespread use of open source technology to power big data initiatives. Open source systems allow companies with even the smallest budgets to integrate a cutting-edge solution into their workplace and more easily use their data to glean valuable insights. Furthermore, as an increasing number of companies begin to rely on hybrid and public cloud platforms for their big data projects, operational costs can be greatly reduced, implementation and integration become almost effortless, and efficiencies and scalability are greatly improved.

However, for many companies, the rise of big data is not an intuitive one. A McKinsey report recognizes that for businesses running legacy technologies with deeply established operational processes, overhauling those systems is almost unimaginable. Others have invested in new technologies, but don’t have the necessary infrastructure, resources or procedures to leverage the true power of those investments. If they are to keep up with the shifting tech landscape, sooner or later they must take a page from leading companies that are using big data to transform their operations and uncover hidden business opportunities.

Of course, as big data is adopted by more and more businesses, there is the inevitable risk of security breaches. This is especially worrisome considering big data’s close connection with the Internet of Things, as a massive number of devices make opportunities for data collection more available. Privacy and security protections should be a vital focus of any big data initiative, but even then, Gartner predicts that by 2018, 50 percent of business ethics violations will occur because of big data misuse.

The Big Data Skills Gap

Several years ago, a McKinsey study predicted that the US would face a big data talent shortage to the tune of 140,000 to 190,000 people versus almost half a million jobs. It’s a report that rings true half a decade later, with 75 percent of companies investing in big data and demand outstripping supply three to one. Furthermore, because big data impacts every area of business, the basic skills needed to work with it are becoming a requirement across functions, not just in the IT department.

Part of the problem is the rapid pace and level of specialization with which big data technology has progressed and made its way into business. The rate at which IT professionals are adopting these skills falls significantly short, often due to a lack of educational resources. Most universities and colleges are well behind the times. The few that do offer big data courses often only make it available to graduate students; far from enough to help fill the demand.

eLearning appears to be the go-to solution for tech professionals looking to develop their big data skills. But even then, the demand for specialized data analytics and management skills is monumental. Careerbuilder reports that the skillset is one of the hardest to recruit for, documenting a massive 77,010 job postings with just 4,356 active candidates in the last six months. Of those candidates, 40 percent have a master’s degree and 42 percent have 6-10 years’ experience. This demographic sweet spot is elusive, and competition for these individuals has hiked compensation up to unprecedented rates, with an average salary of $132,916.

For companies whose budgets can’t stretch to accommodate that expense, big data initiatives may seem out of reach. However, understanding how to build big data skills on top of other foundational technical skills can help leverage existing tech staff or even recent computer science grads. Furthermore, implementing an online training program with courses in Hadoop and other key big data skills can help those individuals gain ground more quickly. Finally, turning to a contingent IT workforce to manage these cutting-edge initiatives can also help businesses actualize high ROI, as well as achieving operational efficiencies and reduced costs.

What does 2017 look like for you? If you’re searching for big data professionals, let us know. We’d love to help.


Brian Knapp is the Executive Vice-President, Operations at MDI Group. If you are interested in learning more about how to justify hiring the best IT talent, contact Brian directly at, or call us 888-416-7949.  MDI Group has offices and specializes in recruiting IT talent in Atlanta, Austin, Charlotte, Dallas/Fort Worth, Greenville and Phoenix.