Senior Data Engineer

Responsibilities:

Build Data Pipelines:

Utilize PySpark and Python to construct efficient and scalable data pipelines.

Integrate data from multiple source systems into a unified target system.

Orchestrate Pipelines with Airflow:

Use Apache Airflow to orchestrate and schedule data pipelines, ensuring timely and reliable execution.

Enhance Existing Pipelines:

Understand existing data pipelines and make enhancements based on evolving business requirements.

Implement improvements to optimize performance and maintainability.

Debugging and Root Cause Analysis:

Troubleshoot and resolve any failures in data pipelines promptly.

Conduct root cause analysis for pipeline failures and implement corrective measures.

Collaboration with Stakeholders:

Work closely with various stakeholders, both within and across teams.

Communicate effectively to understand and address business needs related to data ...