Tech Stack

Description

You will design, build, and optimize high-performance, scalable data pipelines using PySpark and Apache Spark to support advanced analytics and data-driven decision-making. This role involves developing data models, implementing efficient transformations, and collaborating with cross-functional teams to ensure data quality and reliability in large-scale environments.

Requirements

• Strong hands-on experience with Apache Spark, PySpark, and AWS Glue, building scalable data pipelines in distributed environments
• Proven experience designing and maintaining AWS-based data architectures, including S3, Athena, and data warehouse solutions
• Advanced proficiency in Python and SQL, with experience processing and transforming large datasets in relational databases
• Experience with ETL/ELT pipelines, data lakes, and event-driven architectures, ensuring scalable and reliable data workflows
• Strong collaboration, analytical, and problem-solving skills, with experience working in Agile environments and building BI/reporting solutions (e.g., QuickSight)

Responsibilities

• Design, build, and optimize scalable data pipelines using PySpark and Apache Spark for distributed data processing
• Develop and maintain data models, ensuring proper integration of multiple data sources into a centralized data lake and data warehouse
• Implement efficient data transformations and optimize pipelines for performance, scalability, and reliability in large-scale environments
• Create and manage data solutions for analytics, including SQL-based transformations, AWS Athena views, Iceberg tables, and BI dashboards (QuickSight)
• Collaborate with cross-functional teams and ensure data quality through monitoring, troubleshooting, and continuous improvement of data workflows