Lead Data Engineer

Key Responsibilities

• Data Ingestion Framework:

o Design & Development: Architect, develop, and maintain an end-to-end data ingestion framework that efficiently extracts, transforms, and loads data from diverse sources.

o Framework Optimization: Use AWS services such as AWS Glue, Lambda, EMR, ECS , EC2 and Step Functions to build highly scalable, resilient, and automated data pipelines.

• Data Quality & Validation:

o Validation Processes: Develop and implement automated data quality checks, validation routines, and error-handling mechanisms to ensure the accuracy and integrity of incoming data.

o Monitoring & Reporting: Establish comprehensive monitoring, logging, and alerting systems to proactively identify and resolve data quality issues.

• API Development:

o Design & Implementation: Architect and develop secure, high-performance APIs to enable seamless integration of data services with external appl...