Back to SearchLead Big Data Engineer with Python
Sorry, this position is no longer available
We are looking for a highly skilled remote Lead Big Data Engineer with Python to develop, monitor, and operate the most critical curated data pipeline.
The successful candidate will work closely with analysts, data scientists, and product managers to build and continuously improve the "Single Source of Truth" KPI for business steering.
In this position, you will leverage and improve a cloud-based tech stack that includes AWS, Databricks, Kubernetes, Spark, Airflow, Python, and Scala. If you are a highly driven individual with strong technical skills, this is the role for you!
Responsibilities
- Building, monitoring, and operating critical curated data pipelines
- Consulting with analysts, data scientists, and product managers to build and continuously improve the "Single Source of Truth" KPI for business steering
- Redeveloping old legacy pipelines to new, advanced, and standard versions that are easy to maintain and scalable for future demands
- Leverage and improve a cloud-based tech stack that includes AWS, Databricks, Kubernetes, Spark, Airflow, Python, and Scala
- Mentoring and guiding junior team members
Requirements
- 5+ years of experience in data software engineering
- 1+ years of relevant leadership experience
- Expertise in Apache Spark along with Spark streaming
- Good hands-on experience with Databricks and delta-lake
- Fluency in Python programming language
- Good understanding & hands-on experience with CI/CD
- Comfortable working with AWS landscape
- Experience in building Apache Airflow pipelines
- B2+ English level
Nice to have
- Experience in Presto, Superset, Starburst, Oracle & Exasol
Benefits
- International projects with top brands
- Work with global teams of highly skilled, diverse peers
- Healthcare benefits
- Employee financial programs
- Paid time off and sick leave
- Upskilling, reskilling and certification courses
- Unlimited access to the LinkedIn Learning library and 22,000+ courses
- Global career opportunities
- Volunteer and community involvement opportunities
- EPAM Employee Groups
- Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn