Software Data Engineer - MFS - IT - EXEMPT
- Architect, design, develop, implement, and maintain code to support data processing, and flows thru data lake
o Landing Zone - data ingestion of raw data or capture of streaming data
o Discovery Zone - evaluate data quality, transform raw data, and cleanse data
o Enterprise Zone - transform into data model for external consumption by reporting & self-service BI
- Architect, design (including data model), develop, implement, and maintain code for self-service BI and reports
- Analyze and develop designs for enterprise systems to meet business requirements
- Provide software expertise in one or more of these areas: data security, data storage, data ingestion, data streaming, data processing, data analysis & exploration, data quality, data visualization (using reporting & business intelligence), business process management/business rules processing, data exchange, or data modelling.
- Lead or perform proof-of-concept initiatives in new technologies
- Be a contributing member of a cross-functional sprint team comprised of analysts, UX, DevOps, developer, testers, release management, and the product owner
- Support sprint team in acceptance test driven development (ATTD) by building appropriate test automation including unit tests, component tests, integration tests, etc.
- Recommend, design, implement and maintain the various file formats (e.g. XML/XSD, SequenceFiles, Avro files, or Parquet files) that define the information interchange between application, external systems, and/or data lake.
- Investigate and repair application defects regardless of component including data ingestion, data streaming, data transformation, data cleansing, data modelling, reporting, self-service BI, or database.
- Collaborate with business analysts, data analysts, architects to analyze business requirements, develop data model & data capture strategy, and develop technical design specifications
- Support architect and contribute to strategic implementations of enterprise products and solutions using the latest technologies following software engineering best practices
- Assist DevOps team in understanding, maintaining, and improving build scripts & ensure their integration in deployment pipeline
- Gather and assemble information to support risk assessment and security vulnerabilities of client and corporate enterprise systems
- Assemble, deliver and share technical information and training materials used across many technical disciplines and staff roles
- Lead less experienced software engineers in developing technical design specifications
- Debrief senior management and executive staff on technical solutions.
- Provide software development estimates outlining business and technical risks.
- Represent team at technical meetings pertaining to schedule, requirements, project plans, technical decisions, data discrepancies and other technical matters, as directed by team leadership.
- Recommend and enforce policies for the source-code control system (GIT) and the team's use thereof. (e.g. "don't break the build", or "DBTB" - the policy that no check-in should cause a failure in the DevOps pipeline.)
- Contribute to team coding standards and monitor the code base for compliance.
- Contribute to and expand in-house documentation technical or otherwise.
- Provide third level customer support.
- All other duties as assigned or directed
- Bachelor's Degree in Computer Science or a related field
- 6+ years of experience of full lifecycle development
- 6+ years of systems analysis & application design experience
- 2+ years of experience architecting, designing, and developing custom Spark or Hadoop MapReduce applications using Java, or Scala
- 2+ years of experience in designing, building, and using a big data distribution, preferably Cloudera (Hortonworks, or MapR), for
o data ingestion, cleansing, and transformation (e.g. Talend)
o data discovery & analysis using querying tools (e.g. Impala, Hive)
o data storage using distributed databases (HBASE, Kudu)
o data streaming (e.g. Kafka, Apache Spark)
o processing monitoring (e.g. Cloudera manager, Hue)
- Education and/or formal training may substitute for experience requirement
- Ability to communicate clearly and professionally with all levels of the organization, both written and verbal
- Ability to work well in a team environment, to collaborate with others, and interface with team members internal and external to the organization
- Ability to establish and maintain effective professional relationships with internal and external stakeholders
- Must be able to adapt to organizational change
- Must be proficient in Microsoft Office Suite
- Ability to plan, prioritize, and execute multiple tasks in a fast paced environment
- Must be self-motivated, well-organized, and detail oriented
- Experience with formal software processes and lifecycle requirements.
- Ability to assist in the collection and consideration of required information and data.
- Demonstrable knowledge with Java EE, Java generics, lambda functions, and concurrent programming.
- Demonstrable Spring framework.
- Excellent knowledge of relational databases (Oracle, PostgreSQL), SQL and query performance analysis (query analyzer, explain plan).
- Preferred experience with ATTD and associated technologies (Fitnesse, DBSLim, Junit)