Skip To Main Content
epam-logo-black.svgepam-logo-black.svg
backBack to Search

Lead Data Eng Developer

Colombia, Mexico
Data Software Engineering, Amazon Web Services, CI/CD, Python, SQL, Amazon DynamoDB, Apache Hive, Apache Spark, Databricks

We are looking for a highly experienced Lead Data Engineering Developer to join our team. This role involves building, optimizing, and maintaining complex data pipelines, leveraging your expertise in Python, Spark, Hive, and cloud environments. You'll play a critical role in delivering high-quality data solutions for dynamic projects.

Responsibilities
  • Design, develop, and maintain complex data pipelines to support large-scale data processing
  • Write clean, efficient, and test-driven code using Python and SQL
  • Implement unit testing with mock frameworks as part of the development process
  • Work extensively with Spark and Hive for data processing and transformation
  • Collaborate with teams to optimize data structures and algorithms for performance
  • Utilize AWS components (EC2, S3, Lambda, DynamoDB, SQS, RDS) for cloud-based data processing and storage
  • Ensure the scalability and reliability of data workflows and pipelines
Requirements
  • At least 7 years of hands-on experience in data engineering or related fields
  • Strong coding skills in Python with deep expertise in SQL
  • Experience with test-driven development and writing unit tests using mock frameworks
  • Proven ability to design and manage complex data pipelines
  • Extensive knowledge of Spark and Hive data processing frameworks
  • Solid understanding of data structures and algorithms
  • Hands-on experience with AWS cloud services including EC2, S3, Lambda, DynamoDB, SQS, and RDS
  • English language proficiency at B2+ level
Nice to have
  • Experience with data orchestration tools like Airflow
  • Familiarity with other cloud platforms or additional AWS services
  • Background in optimizing data pipelines for machine learning or analytics use cases
Benefits
  • International projects with top brands
  • Work with global teams of highly skilled, diverse peers
  • Healthcare benefits
  • Employee financial programs
  • Paid time off and sick leave
  • Upskilling, reskilling and certification courses
  • Unlimited access to the LinkedIn Learning library and 22,000+ courses
  • Global career opportunities
  • Volunteer and community involvement opportunities
  • EPAM Employee Groups
  • Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn