Big Data Hadoop Architect Lead

  • location: New York, NY
  • type: Permanent
  • salary: $150,000 - $180,000 per year
easy apply

job description

Big Data Hadoop Architect Lead

job summary:
Candidates must have proper work authorization! Cannot provide sponsorship at this time.

The Big Data Hadoop Architect / Lead position will be part of the Insight Core Hadoop platform team within Global Banking and Markets. The role is expected to lead deliverables around platform design & configuration, capacity planning, incident management, monitoring, and business continuity planning

 
location: New York, New York
job type: Permanent
salary: $150,000 - 180,000 per year
work hours: 9am to 6pm
education: Bachelors
 
responsibilities:
Responsibilities

-Responsible for developing, enhancing, modifying and/or maintaining a multi-tenant big data platform -Functionally lead a team of developers located on and off shore and collaborate with Product Owners, Quants, and other technology teams to deliver data/applications/tools

-Work closely with the Business Stakeholders, Management Team, Development Teams, Infrastructure Management and support partners

-Use your in-depth knowledge of development tools and languages towards design and development of applications to meet complex business requirements

-Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc

-Design and implement scalable data platforms for our customer facing services

-Deploy and scale Hadoop infrastructure

-Hadoop / HDFS maintenance and operations

-Data cluster monitoring and troubleshooting

-Hadoop capacity planning

-OS integration and application installation

-Partner with program management, network engineering, site reliability operations, and other related groups

-Willingness to participate in a 24x7 on-call rotation for escalations

 
qualifications:
-Deep understanding of UNIX and network fundamentals

-Expertise with Hadoop and its ecosystem Hive, Pig, Spark, HDFS, HBase, Oozie, Sqoop, Flume, Zookeeper, Kerberos, Sentry, Impala etc.

-Experience designing multi-tenant, containerized Hadoop architectures for memory/CPU management/sharing across different LOBs

-5+ years managing clustered services, secure distributed systems, production data stores

-3+ years of experience administering and operating Hadoop clusters

-Cloudera CHD4 /CDH5 cluster management and capacity planning experience

-Ability to rapidly learn new software languages, frameworks and APIs quickly

-Experience scripting for automation and config management (Chef, Puppet)

-Multi-datacenter, multi-tenant deployment experience, a plus

-Strong troubleshooting skills with exposure to large scale production systems

-Hands on development experience and high proficiency in Java / Python

-Skilled in data analysis, profiling, data quality and processing to create visualizations

-Experience working with Agile Methodology

-Good SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases (Hive, Impala, Kudu a plus).

-Experience building and optimizing 'big data' data pipelines, architectures and data sets.

-Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

-Strong analytic skills related to working with unstructured datasets.

-Build processes supporting data transformation, data structures, metadata, dependency and workload management.

-Experience supporting and working with cross-functional teams in a dynamic environment.

 
skills: Must have skills:

  • Deep understanding of UNIX and network fundamentals
  • Expertise with Hadoop and its ecosystem Hive, Pig, Spark, HDFS, HBase, Oozie, Sqoop, Flume, Zookeeper, Kerberos, Sentry, Impala etc.
  • Experience designing multi-tenant, containerized Hadoop architectures for memory/CPU management/sharing across different LOBs
  • 5+ years managing clustered services, secure distributed systems, production data stores
  • 3+ years of experience administering and operating Hadoop clusters
  • Cloudera CHD4 /CDH5 cluster management and capacity planning experience
Educational Requirements:

-Bachelor's Degree in Information/Computer Science or related field OR equivalent professional experience


Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

easy apply

get jobs in your inbox.

sign up
{{returnMsg}}

related jobs

    Senior Data Architect

  • location: New York, NY
  • job type: Contract
  • salary: $67 - $77 per hour
  • date posted: 10/18/2019

    Lead Java/Scala Developer

  • location: New York, NY
  • job type: Permanent
  • salary: $130,000 - $180,000 per year
  • date posted: 10/17/2019