Job Description

Exigo Tech is a Sydney-based Technology Solutions Provider that is focused on providing solutions on three major verticals; Infrastructure, Cloud, and Application to businesses across Australia. We help companies reach operational efficiencies by empowering them with technology solutions that drive their business processes.

Exigo is looking for Full-time Sr. Data Engineer

Roles and Responsibilities

  • Install, configure, and manage Apache Spark (open-source) clusters on Ubuntu, including Spark master/worker nodes and Spark environment files.
  • Configure and manage Spark UI and Spark History Server for monitoring jobs, analyzing DAGs, stages, tasks, and troubleshooting performance.
  • Develop, optimize, and deploy PySpark ETL/ELT pipelines using DataFrame API, UDFs, window functions, caching, partitioning, and broadcasting.
  • Deploy PySpark jobs using spark-submit in client/cluster mode with prope...

Ready to Apply?

Take the next step in your AI career. Submit your application to Exigo Tech today.

Submit Application