Skip To Main Content
backBack to Search

Lead Data Software Engineer

Hybrid in Portugal
Data Software Engineering
& 6 others

We are seeking a talented and motivated Lead Data Software Engineer to join our team.

In this role, you will leverage deep expertise in cloud-native technologies like Databricks, Azure DevOps, Delta Lake, and Spark Structured Streaming within a Microsoft Azure environment to lead and guide teams in designing, implementing, and maintaining scalable, performant, and transparent data-processing pipelines for critical financial data. You will take ownership of technical initiatives, drive thought leadership, and push the boundaries of system scalability and efficiency while fostering cross-functional collaboration.

If you are ready to take on a leadership role in a collaborative and innovative environment, we want to hear from you!

Office presence of about 3 days a week is desirable.

Responsibilities
  • Lead the design, development, maintenance, and optimization of efficient and scalable data pipelines using Python and Apache Spark
  • Architect streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming, ensuring high system performance and dependability
  • Collaborate closely with business partners, IT experts, architects, and engineers to gather requirements, define project goals, and design strategies
  • Mentor and support team members through code reviews, technical training, and collaborative problem-solving
  • Utilize relational data models and SQL to manage, query, and finesse large-scale data workflows
  • Develop and enforce automated testing strategies to ensure high-quality, reliable, and maintainable solutions
  • Monitor, troubleshoot, and anticipate complex data processing issues while implementing proactive measures to maintain operational stability across systems
  • Document technical designs, processes, and architecture best practices with precision and forward-thinking clarity
  • Identify and drive initiatives for continuous improvement, from operational efficiency to technical scalability, within data pipelines and team processes
  • Engage with product owners, senior leadership, and team members to prioritize and deliver impactful tools and solutions aligned with business goals
  • Take ownership of technical challenges and act as a primary point of contact for critical decisions and escalations within data engineering initiatives
Requirements
  • 5+ years of experience in a software engineering or data engineering role working with large-scale distributed data systems
  • At least 1 year of relevant leadership experience
  • Advanced expertise in Python, Apache Spark, and Delta Lake
  • Deep familiarity with relational data models, SQL, and complex data transformations
  • Proven mastery of cloud-native technologies within the Microsoft Azure ecosystem, including Databricks and Azure DevOps
  • Strong experience in designing and maintaining reliable automated testing frameworks for data solutions
  • Exceptional analytical skills to break down complex problems, design actionable strategies, and deliver elegant solutions
  • Demonstrated ability to mentor and coach team members to elevate overall technical and professional outcomes
  • Fluency in English and excellent communication skills to lead and collaborate effectively with distributed cross-functional teams
Nice to have
  • Deep understanding of Spark Structured Streaming for real-time and near real-time data processing use cases
  • Significant experience working with financial data or similarly complex data domains requiring high accuracy and efficiency
  • Proven track record in identifying and resolving performance bottlenecks in distributed systems, with a focus on both technical and organizational solutions
  • Strong proficiency in documenting long-term data pipeline architecture and implementation approaches, specifically tailored for scalability and best practice alignment
  • Prior experience in leading technical decision-making and steering large-scale data engineering projects to successful completion
Benefits
  • International projects with top brands
  • Work with global teams of highly skilled, diverse peers
  • Healthcare benefits
  • Employee financial programs
  • Paid time off and sick leave
  • Upskilling, reskilling and certification courses
  • Unlimited access to the LinkedIn Learning library and 22,000+ courses
  • Global career opportunities
  • Volunteer and community involvement opportunities
  • EPAM Employee Groups
  • Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn