Demand For Data Engineers Exceeds Supply
By Christine Kern, contributing writer
Companies face a data engineering talent vacuum.
Data engineers are becoming the most in-demand tech employees and, according to one report, there aren’t nearly enough of them. Software creates valuable data — information on customers, locations, machine efficiency, even the temperature of your coffee — and companies are struggling to figure out how to use it all. One of the biggest challenges is finding the right experts who can interpret the mountain of data.
The New York Times illustrates this point in a story on General Electric in which CEO Jeff Immelt “was speaking with G.E. scientists about new jet engines they were building, laden with sensors to generate a trove of data from every flight — but to what end? That data could someday be as valuable as the machinery itself, if not more so. But G.E. couldn’t make use of it.”
Making use of data is what data engineers do. As more software creates more data, companies need more data engineers. But, according to a report released by Stitch and Galvanize, there are only 6,500 self-reported data engineers on LinkedIn, but more than 6,600 job openings in the San Francisco Bay area alone. And while the number of data engineers more than doubled between 2013 and 2015, the demand for their services also has skyrocketed.
In January, Robert Half Technology released its 2016 Salary Guide for Technology Professionals and found there is still more demand than supply for IT professionals, creating a competitive recruiting environment. That’s a massive gap and it’s only going to grow.
The need for data talent is born from a fundamental shift: tech companies are now data companies. Jake Stein, CEO of Stitch, explained, “Companies are increasingly viewing data as a competitive advantage. The Ubers and Airbnb’s of tech have mastered using data to build better and smarter technology products. Now other tech companies are looking to do the same, and this is causing a major talent shortage.”
The study analyzed 30,000 professional experience records and 82,000 skills records at nearly 3,500 companies, finding the information technology and services industry employs the largest number of data engineers. This includes G.E. which, according to Immelt has “probably hired, since we started this, a couple thousand data scientists and people like that. That’s going to continue to grow and multiply.”
“The need for data science skills has become dramatic because companies realize the value and growth potential in their data assets,” said Jim Deters, co-founder and CEO of Galvanize.