Data Engineer

• Design, develop, maintain efficient and scalable solutions using PySpark
• Ensure data quality and integrity by implementing robust testing, validation and cleansing processes
• Integrate data from various sources, including databases, APIs, external datasets etc.
• Optimize and tune PySpark jobs for performance and reliability
• Document data engineering processes, workflows and best practices
• Strong understanding of databases, data modelling, and ETL tools and processes
• String programming skills in python and proficiency with PySpark, SQL
• Experience with relational databases, Spark, AWS, Python skill
• Excellent communication and collaboration skills

Key Responsibilities:

• Design and Development: Create, develop, and maintain robust solutions using PySpark to handle large-scale data processing.
• Data Quality Assurance: Implemen...