Job Description
Streaming & Real-Time Data Processing
- Design and develop real-time streaming pipelines using Databricks Structured Streaming.
- Build and maintain Kafka-based ingestion frameworks.
- Handle late-arriving events using watermarks, event-time processing, and stateful streaming concepts.
- Implement exactly-once processing and checkpointing mechanisms.
- Monitor and optimize streaming workloads for performance and reliability.
Spark Declarative Pipelines (SDP) & Delta Live Tables (DLT)
- :
Design and implement Spark Declarative Pipelines using Databrick - s.Develop Delta Live Table (DLT) pipelines for scalable data transformation
- s.Implement data quality expectations, validations, retries, and failure handling within DLT pipeline
Ready to Apply?
Take the next step in your AI career. Submit your application to Xebia today.
Submit Application