Responsible for building, operationalizing and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. You should be a skillful data pipeline builder and data wrangler who has experience optimizing building data systems from the ground up. You are expected to support our software developers, database architects, data analysts and data scientists on data initiatives and ensure optimal data delivery architecture is consistent throughout ongoing projects.
location: San Francisco, California
job type: Contract
salary: $80 - 90 per hour
work hours: 8am to 5pm
education: Bachelors
responsibilities:
You must have excellent communication skills, have a strong sense of ownership, be self-directed and comfortable supporting the data needs of multiple cross-functional teams.
This is a hands-on role and requires prior programming experience working with data on cloud.
Responsibilities
Data Pipeline Building, Optimization and Operationalization
? Create and maintain optimal data pipeline architecture,
? Assemble large, complex data sets that meet functional / non-functional business requirements.
? Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability,
? Operationalizing, Improving the quality and reliability of data pipelines (monitoring, retry, failure detection)
Data Platform and Self-serving Data Infrastructure
? Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using various data tools and cloud services.
? Design and build abstractions that hide the complexity of the underlying big data processing and execution stack and that allow data consumers/partners to focus on their strengths: product, data modeling, data analysis, search, information retrieval, and machine learning.
? Build processes and infrastructure to manage lifecycle of datasets: data structures, metadata, dependency and workload management.
? Building internal services and tools to help in-house partners implement, deploy and analyze datasets with a high level of autonomy and limited friction.
Qualifications
? MS or BS in CS, Engineering, Math, Statistics, or a related field or equivalent practical experience in data engineering.
? 3-5 years of experience working in data engineering using Python on Cloud platforms like AWS. Expert in programming in Python. With experience of packages like pandas, numpy, boto, s3fs, flask, fastapi, pyarrorw, pyspark etc.
? Background in software development or engineering
? Working experience with open-source platforms for data engineering pipelines in AWS mainly with Apache Airflow.
? Working experience with Data Processing methods and techniques which includes Batch processing, online processing etc..
? Working experience with distributed computing framework and set of libraries for real-time, large-scale data processing using PySpark.
? Working experience with Agile practices includes requirements discovery and solutions improvement through the collaborative effort of self-organizing and cross-functional teams.
? Working experience with CI/CD using AWS infrastructure and services.
? Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
? strong sense of ownership.
? Collaborative personality with experience supporting and working with cross-functional teams in a dynamic environment.
qualifications:
- Experience level: Experienced
- Minimum 8 years of experience
- Education: Bachelors
skills:
Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.
Qualified applicants in San Francisco with criminal histories will be considered for employment in accordance with the San Francisco Fair Chance Ordinance.
We will consider for employment all qualified Applicants, including those with criminal histories, in a manner consistent with the requirements of applicable state and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance.
For certain assignments, Covid-19 vaccination and/or testing may be required by Randstad's client or applicable federal mandate, subject to approved medical or religious accommodations. Carefully review the job posting for details on vaccine/testing requirements or ask your Randstad representative for more information.
