Back to SearchLead Data DevOps Engineer
Data DevOps
& 3 others
Choose an option
We are looking for a Lead Data DevOps Engineer to drive the migration of Spark, Flink, and additional services to Databricks.
You will be responsible for creating and managing robust cloud infrastructure to enhance data processing pipelines. Join us to leverage your AWS, Terraform, and Databricks expertise and help deliver cutting-edge data solutions.
Responsibilities
- Develop, launch, and oversee Databricks infrastructure with Terraform to support scalable data operations
- Enhance Spark and Flink workflows and coordinate job orchestration for optimal pipeline performance within Databricks
- Architect, implement, and sustain AWS cloud infrastructure ensuring high availability and top performance
- Work closely with stakeholders to collect requirements and execute infrastructure upgrades aligned with business goals
- Track and resolve issues in data infrastructure using monitoring tools like Datadog to guarantee uptime
- Refine and stabilize CI/CD pipelines leveraging Spinnaker and Jenkins for seamless continuous delivery
- Maintain adherence to security policies and best practices across the data infrastructure
Requirements
- Demonstrated track record in Data DevOps engineering with over 5 years managing cloud infrastructure
- Advanced skills in Amazon Web Services (AWS) cloud environment
- Proficient in managing Databricks platforms and associated data workflows
- Experienced in using Terraform for infrastructure as code deployment and management
- Comprehensive understanding of Spark and Flink data processing systems
- Capability to enhance CI/CD pipelines and automate deployment workflows
- Effective communication skills to collaborate within cross-functional teams
- Upper-Intermediate English proficiency (B2 level) for effective interaction with international teams