Job Description
Description
:
Responsibilities:
Develop & Optimize Data Pipelines Build, test, and maintainETL/ELT data pipelines using AzureDatabricks & Apache Spark (PySpark) .Optimizeperformance and cost-efficiency of Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverless Implement Unity Catalog for Data Governance Design and enforceaccess control policies using Unity Catalog.Managedata lineage, auditing, and metadata governance .Enable secure data sharing across teams and external stakeholders. Integrate with Cloud Data Platforms Work withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams .ImplementDelta Lake for scalable, ACID-compliant storage. Automate ...
Ready to Apply?
Take the next step in your AI career. Submit your application to Toppan Merrill today.
Submit Application