Back to SearchLead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes
Remote in Georgia, & 4 others
Data Software Engineering
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a job
Choose an option
We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.
Responsibilities
- Implement streaming and batch Spark pipelines running on Kubernetes
- Lead the implementation of objects within the data generator framework
- Oversee deployment and testing across local and development environments
- Drive improvement of the current, actively changing solution
- Conduct bug investigation and resolution
- Perform unit testing to ensure solution quality
- Participate in refinement, planning, and demo sessions
- Guide and mentor team members throughout the implementation phase
Requirements
- 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes
- Proficiency in Python
- Familiarity with the data engineering domain and cloud-agnostic platform migrations
- Capability to dive deeper into new technologies in a short time during active implementation phases
- Ability to work independently with syncs with a team lead
- Strong communication skills and proactiveness
- Showcase of being a hands-on team player and technical leader
- Proficiency in English at a B2+ level
Nice to have
- Knowledge of Java
- Familiarity with Kafka Connect, KSQL, and Kafka Streams
- Skills in Ansible and Argo CD
- Understanding of TDD