One of Randstad Technologies’ hot jobs is big data engineer
. There can be confusion about the specific roles people with this title fill. According to the big data news portal, Datanami.com
, data engineers are the specialists who are the bridge between the data analysts and the data scientists. They are the people charged with providing the data analytics applications and help obtain quality data from different sources in the appropriate format(s) and make sure that users have access to that data quickly. They are adept at manipulating enormous quantities of data and making sure that the organization’s big data software stack can scale to support the workload.
As with all other Randstad Technologies’ Hot Jobs
(full stack developers
, IoT architects
, VR engineers and security engineers
), the demand for big data engineers is strong and will only get stronger. According to a report written by Stitch
, an ETL service built for developers, there are fewer than 7,000 self-reported data engineers in the U.S. The same report notes that at the time it was written there were 6,600 job openings for data engineers
-- in the San Francisco Bay Area alone! In the past 30 days
there have been more than 1,000 jobs posted on Monster.com with either “data engineer” or “big data engineer” in the title. IBM, Cisco
and Oracle posted a combined total of nearly 27,000 open jobs that required big data expertise in the last 12 months. Finding the people to fill all these jobs will be challenging.
Profile of big data engineers
Overview of data engineer characteristics/positions:
- Around 40 percent of data engineers graduated from a software engineering role. Other positions from which data engineers come include analyst, consultant, business analyst, data architect, data analyst, database administrator and data scientist.
- Some of the most important technical skills for data engineers to possess are SQL, Java, Hadoop, Python, Linux, Spark and Hive.
- Annual salaries for data engineers have been trending upward of late, with the average ranging between $100,000 and $200,000 and the median at $141,000.
- Of those nearly 7,000 people who classify themselves as data engineers, 40 percent have a bachelor’s degree, 27 percent a master’s degree and 18 percent an associate degree, with around 4 percent having a Ph.D. Sixty-one percent of big data engineer openings require a bachelor’s degree.
- There are a number of certification programs, some of which are offered by universities (e.g., Certification of Professional Achievement in Data Sciences from Columbia University) and some by private firms (e.g., Cloudera Certified Data Engineer, Galvanize Spark workshops, etc.). Price and duration of each program can vary greatly.
- There are various estimates, but it appears that men outnumber women as data engineers from around 5 to 1 to as much as 7 to 1.
Who hires big data engineers?
According to Forbes
, the top five industries hiring big data-related expertise include professional, scientific and technical services (30 percent), information technologies (19 percent), manufacturing (18 percent), finance and insurance (10 percent), and retail trade (eight percent). Big employers include Google, Uber, Facebook, Amazon, Verizon, Capital One, AirBnB, IBM, Cisco, Oracle, EMC
and Spotify, among others. Given that the magnitude and value of data is
going to continue to grow at an accelerating pace, even smaller companies are going to have to employ data engineers to find ways to effectively acquire, organize and analyze their data if they are to remain competitive.
Soft skills a plus
As with almost all technical positions, having soft skills enhances a candidate’s attractiveness and ability to fashion their own future. Because data engineers often have to explain highly technical concepts and findings to non-tech personnel, strong communications skills are highly attractive.
The demand for data engineers is going to outstrip supply for some time. Competition for individuals with the necessary skills will remain intense for the foreseeable future. It will remain one of the hottest of our hot jobs.
Looking for a new career opportunity as a big data engineer? Search our website today for big data engineer jobs in your area.