Tech Stack

Description

You will design, build, and maintain scalable data pipelines and ETL processes on cloud platforms, collaborating with AI engineers and data scientists to support model development and production services. You'll optimize performance, enforce data governance, and evaluate new technologies for generative AI and machine learning applications.

Requirements

• Bachelor's degree in Computer Science, Engineering, or related technical field
• 6+ years professional experience in data engineering
• Proficient in Python for production data systems
• Strong experience with Spark or similar distributed computing frameworks
• Experience building data pipelines on AWS, strong SQL, and data modeling skills

Responsibilities

• Design, build, and maintain scalable data pipelines and ETL processes using Python, Spark, and Databricks on AWS
• Ingest, normalize, and enrich large volumes of structured and unstructured data
• Implement automated data validation, testing, monitoring, and alerting for data quality and drift detection
• Optimize pipeline performance and cost through distributed compute and cloud resource management
• Enforce data governance, access controls, and privacy standards while maintaining metadata and data lineage