Intermediate - Senior Data Engineer (Databricks)

We are looking for a Data Engineer who is certified in Databricks (required) to join our team. In this role you will be designing, developing, and optimizing scalable data pipelines and workflows on Databricks. The engineer will work closely with stakeholders to make certain data reliability, performance, and alignment with business requirements.

Scope of Work

Data Pipeline Development:

• Building efficient ETL/ELT pipelines using Databricks and Delta Lake for structured, semi-structured, and unstructured data.
• Transforming raw data into consumable datasets for analytics and machine learning.

Data Optimization:

• Improving performance by implementing best practices like partitioning, caching, and Delta Lake optimizations.
• Resolving bottlenecks and ensuring scalability.

Data Integration:

• Integrating data from various sources such as APIs, databases, and cloud storage systems (e.g., AWS S3, Azu...