Data Engineer Databricks Dataiku (Freelance optional)

Tasks and responsibilities

Our client works as an Agile organisation.You will be dedicated to a large transformation project, with other Databricks/Dataiku developers.

You demonstrate and use your developer experience to develop and maintain complex data transformation pipelines under specific guidelines

You are familiar of working as data engineer in Databricks (on top of AWS)
Your responsibilities will be to:
• Design, develop and maintain scalable data pipelines
• Manage data ingestion processes from various sources (xml ingestion, Kakfa, etc.).
• Implement and optimize ETL (Extract, Transform, Load) processes to ensure efficient data transformation and integration.
• Ensure data quality, integrity, and security across all data pipelines.
• Monitor and troubleshoot data pipeline performance, implementing necessary maintenance and updates.
• Document data pipeline processes, technical specifications and best practices.
• Realize know...