Lead Data Engineer 85103

Positions:1 Full Time

Experience

8 - 14 Years

Compensation Details

Variable pay: 10% performance bonus over and above fixed salary

Job Description

Data Pipeline Development & Operations

• Design, build, and operate scalable and reliable data pipelines on the Databricks platform

• Develop end-to-end data workflows from ingestion through transformation to consumption

• Implement robust error handling, monitoring, and alerting mechanisms

• Ensure data pipeline reliability, performance, and maintainability

• Optimize pipeline performance through efficient Spark job design and cluster configuration

• Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

• Refactor legacy code and data pipelines to PySpark for improved performance and scalability

• Migrate traditional ETL p...