AI Data Engineer

Responsibilities

As a data engineer for our advanced analytics platforms, your main responsibilities will be:

Design, test, deploy and maintain production big-data ingestion pipelines using established frameworks, patterns of practice, agile software development and CI/CD practices, working closely with the Principal Software Engineer – Data Ingestion

Work with cross-organizational data source teams to define data ingestion requirements for structured, unstructured and semi-structured data, pilot their implementation, ensure the data source teams accept the resulting landed data as valid

Define and implement automated validation and profiling capabilities needed to ensure reliable data delivery, using agile software development and CI/CD practices

Work with data source teams, domain experts and data scientists to define data cleansing and data enrichment requirements for landed data

Implement data cleansing and enrichment code using established patterns of practice General - Corning (L4)

Work with data source teams, domain experts and data scientists to validate landed, cleansed and enriched data, using agile software development and CI/CD practices, while ensuring that the final datasets are directly usable by them without additional processing effort

Actively participate in code reviews and technical information sharing with your team members and the broader software engineering community at Corning

Stay up to date with industry standards and technological advancements that will improve the quality, productivity and performance of your work.

Provide support in a DevOps environment to monitor tokens, jobs and overall system performance. Qualifications

Bachelor's degree in computer science, engineering, mathematics, or a related technical discipline

Understand concepts of big data engineering, developing and maintaining ETL and ELT pipelines for data warehousing, on-premise and cloud data lake environments

Demonstrated production programming proficiency in at least one modern JVM language such as Java, Scala or Kotlin, as well as an interpreted declarative programming language such as Python

Entry level experience with AWS platform services, including AWS S3 & EC2, Data Migration Services (DMS), RDS, EMR, RedShift, Lambda, DynamoDB, CloudWatch, CloudTrail

Hands-on technical familiarity with Apache Spark architecture, S3, parquet and Delta Lake architecture, technologies and tools

Basic proficiency with both traditional relational and polyglot persistence technologies

Familiarity with agile software development & continuous integration + continuous deployment methodologies along with supporting tools such as Git (Gitlab), Jira, Terraform, New Relic

Familiarity with notebook environments including JupyterHub

Able to Communicate well with users, other technical team to collect requirements, describe data modeling decisions.