Job Description
Job Title: Senior Data Engineer
Key Responsibilities:
Design, develop and maintain scalable data pipelines using Py Spark.
Build and manage batch and real-time data processing systems.
Develop and integrate Kafka-based streaming solutions. * Strong hands-on experience with Py Spark/Apache Spark; Solid understanding of distributed data processing concepts; * Experience working with large-scale datasets;
Aleatory variations:Prior knowledge in containerization tools (Docker, Kubernetes) ,might be useful to ensure the reliability,dependency management,development deployment process.The understanding of workflow orchestration (Airflow/Oozie) could also make a big difference when building these
Key Responsibilities:
Design, develop and maintain scalable data pipelines using Py Spark.
Build and manage batch and real-time data processing systems.
Develop and integrate Kafka-based streaming solutions. * Strong hands-on experience with Py Spark/Apache Spark; Solid understanding of distributed data processing concepts; * Experience working with large-scale datasets;
Aleatory variations:Prior knowledge in containerization tools (Docker, Kubernetes) ,might be useful to ensure the reliability,dependency management,development deployment process.The understanding of workflow orchestration (Airflow/Oozie) could also make a big difference when building these
Ready to Apply?
Take the next step in your AI career. Submit your application to Anonymous today.
Submit Application