Software Data Engineer (Big Data)

  • location: Owings Mills, MD
  • type: Contract
  • salary: $60 - $64 per hour


Software Data Engineer - MFS - IT - EXEMPT

- Architect, design, develop, implement, and maintain code to support data processing, and flows thru data lake

o Landing Zone - data ingestion of raw data or capture of streaming data

o Discovery Zone - evaluate data quality, transform raw data, and cleanse data

o Enterprise Zone - transform into data model for external consumption by reporting & self-service BI

- Architect, design (including data model), develop, implement, and maintain code for self-service BI and reports

- Analyze and develop designs for enterprise systems to meet business requirements

- Provide software expertise in one or more of these areas: data security, data storage, data ingestion, data streaming, data processing, data analysis & exploration, data quality, data visualization (using reporting & business intelligence), business process management/business rules processing, data exchange, or data modelling.

- Lead or perform proof-of-concept initiatives in new technologies

- Be a contributing member of a cross-functional sprint team comprised of analysts, UX, DevOps, developer, testers, release management, and the product owner

- Support sprint team in acceptance test driven development (ATTD) by building appropriate test automation including unit tests, component tests, integration tests, etc.

- Recommend, design, implement and maintain the various file formats (e.g. XML/XSD, SequenceFiles, Avro files, or Parquet files) that define the information interchange between application, external systems, and/or data lake.

- Investigate and repair application defects regardless of component including data ingestion, data streaming, data transformation, data cleansing, data modelling, reporting, self-service BI, or database.

- Collaborate with business analysts, data analysts, architects to analyze business requirements, develop data model & data capture strategy, and develop technical design specifications

- Support architect and contribute to strategic implementations of enterprise products and solutions using the latest technologies following software engineering best practices

- Assist DevOps team in understanding, maintaining, and improving build scripts & ensure their integration in deployment pipeline

- Gather and assemble information to support risk assessment and security vulnerabilities of client and corporate enterprise systems

- Assemble, deliver and share technical information and training materials used across many technical disciplines and staff roles

- Lead less experienced software engineers in developing technical design specifications

- Debrief senior management and executive staff on technical solutions.

- Provide software development estimates outlining business and technical risks.

- Represent team at technical meetings pertaining to schedule, requirements, project plans, technical decisions, data discrepancies and other technical matters, as directed by team leadership.

- Recommend and enforce policies for the source-code control system (GIT) and the team's use thereof. (e.g. "don't break the build", or "DBTB" - the policy that no check-in should cause a failure in the DevOps pipeline.)

- Contribute to team coding standards and monitor the code base for compliance.

- Contribute to and expand in-house documentation technical or otherwise.

- Provide third level customer support.

- All other duties as assigned or directed


- Bachelor's Degree in Computer Science or a related field

- 6+ years of experience of full lifecycle development

- 6+ years of systems analysis & application design experience

- 2+ years of experience architecting, designing, and developing custom Spark or Hadoop MapReduce applications using Java, or Scala

- 2+ years of experience in designing, building, and using a big data distribution, preferably Cloudera (Hortonworks, or MapR), for

o data ingestion, cleansing, and transformation (e.g. Talend)

o data discovery & analysis using querying tools (e.g. Impala, Hive)

o data storage using distributed databases (HBASE, Kudu)

o data streaming (e.g. Kafka, Apache Spark)

o processing monitoring (e.g. Cloudera manager, Hue)

- Education and/or formal training may substitute for experience requirement


- Ability to communicate clearly and professionally with all levels of the organization, both written and verbal

- Ability to work well in a team environment, to collaborate with others, and interface with team members internal and external to the organization

- Ability to establish and maintain effective professional relationships with internal and external stakeholders

- Must be able to adapt to organizational change

- Must be proficient in Microsoft Office Suite

- Ability to plan, prioritize, and execute multiple tasks in a fast paced environment

- Must be self-motivated, well-organized, and detail oriented

- Experience with formal software processes and lifecycle requirements.

- Ability to assist in the collection and consideration of required information and data.

Technical Skills

- Demonstrable knowledge with Java EE, Java generics, lambda functions, and concurrent programming.

- Demonstrable Spring framework.

- Excellent knowledge of relational databases (Oracle, PostgreSQL), SQL and query performance analysis (query analyzer, explain plan).

- Preferred experience with ATTD and associated technologies (Fitnesse, DBSLim, Junit)

get jobs in your inbox.

sign up

related jobs

    Jr. Web Developer

  • location: Owings Mills, MD
  • job type: Contract
  • salary: 50 - 54 per hour
  • date posted: 11/20/2017

    Full Stack Java Engineer

  • location: Owings Mills, MD
  • job type: Contract
  • salary: 60 - 64 per hour
  • date posted: 11/30/2017