Job Description

Description

:

Responsibilities:

  • Develop & Optimize Data PipelinesBuild, test, and maintainETL/ELTdata pipelines using AzureDatabricks & Apache Spark (PySpark).Optimizeperformance and cost-efficiencyof Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverless
  • Implement Unity Catalog for Data GovernanceDesign and enforceaccess control policiesusing Unity Catalog.Managedata lineage, auditing, and metadata governance.Enable secure data sharing across teams and external stakeholders.
  • Integrate with Cloud Data PlatformsWork withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hubto integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams.ImplementDelta Lakefor scalable, ACID-compliant storage.
  • Automate & Orchestrate Workf...

    • Ready to Apply?

    Take the next step in your AI career. Submit your application to Toppan Merrill today.

    Submit Application