Back to SearchSenior Data Engineer (Spark + AWS)
Data Software Engineering, Apache Airflow, Big Data, Kafka Streaming and Connect, Kubernetes, NoSQL in Data, Python, SQL, Spark Streaming, Amazon Web Services
Sorry, this position is no longer available
We are looking for a skilled Senior Data Engineer with experience in Spark and AWS who can design, develop, and maintain data pipelines in our project. The ideal candidate must have an understanding of structured and unstructured data, with advanced skills in Spark, SQL, and No-SQL. They should also have experience in Apache Airflow, Kafka Streaming and Connect, and Kubernetes.
Responsibilities
- Design, develop and maintain data pipelines
- Delivers robust, scalable, and reusable solutions
- Automate data extraction, transformation, and provisioning irrespective of the type of source and target
Requirements
- Understanding of structured & unstructured data, big data
- Knowledge of Core Engineering (versioning, CI/CD, quality, automated testing)
- Experience with Kubernetes (high level), Docker
- SQL Advanced complex data merging techniques including windowing functions etc.
- Understanding of Big-data Concepts (HDFS and overall distributed architecture understanding)
- Work experience in one NoSQL database
- Strong experience with Spark, Spark Streaming, and Performance Tuning in Spark
- Advanced Python programming concepts
- Familiar with Airflow - Managed Airflow Platform
- AWS Cloud Concepts knowledge
- Experience with Snowflake data warehouse
Nice to have
- Good experience in test automation and relative tools and frameworks, like Pytest, Behave, Lettuce, etc.
Benefits
- International projects with top brands
- Work with global teams of highly skilled, diverse peers
- Healthcare benefits
- Employee financial programs
- Paid time off and sick leave
- Upskilling, reskilling and certification courses
- Unlimited access to the LinkedIn Learning library and 22,000+ courses
- Global career opportunities
- Volunteer and community involvement opportunities
- EPAM Employee Groups
- Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn