Data Engineer 93103

Positions:2 Full Time
Experience
6 - 10 Years

Data Engineer JD

Experience required :5 + Relevant

Description

Data Pipeline Development & Operations

• Design, build, and operate scalable and reliable data pipelines on the Databricks platform

• Develop end-to-end data workflows from ingestion through transformation to consumption

• Implement robust error handling, monitoring, and alerting mechanisms

• Ensure data pipeline reliability, performance, and maintainability

• Optimize pipeline performance through efficient Spark job design and cluster configuration

• Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

• Refactor legacy code and data pipelines to PySpark for improved performance and scalability

• Migrate traditional ETL processes to modern ELT patterns on Databricks

• Assess existing codebases and i...