Senior Data Software Engineer
Hybrid in Portugal
Data Software Engineering
& 6 others
Portugal
We are seeking a talented and driven Senior Data Software Engineer to join our team.
In this role, you will harness your expertise in cloud-native technologies like Databricks, Azure DevOps, Delta Lake, and Spark Structured Streaming within a Microsoft Azure environment. You will play a pivotal part in designing, implementing, and maintaining scalable, performant, and transparent data-processing pipelines for critical financial data, pushing the boundaries of system scalability and efficiency while collaborating across teams.
If you are eager to take on an exciting challenge in a collaborative and innovative environment, we want to hear from you!
Office presence of about 3 days a week is desirable.
Responsibilities
- Develop, maintain, and optimize efficient and scalable data pipelines using Python and Apache Spark
- Build streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming
- Collaborate closely with business partners, IT experts, and architects to gather requirements, design solutions, and refine processes
- Participate in peer code reviews to ensure code quality and adherence to best practices
- Utilize relational data models and SQL to manage, query, and process data effectively
- Implement automated testing strategies for data solutions to ensure high reliability and maintainability
- Monitor, troubleshoot, and resolve complex data processing issues to support operational stability
- Document technical designs, processes, and decisions with clarity and precision
- Proactively identify areas for improvement, recommending and implementing solutions
- Engage with product owners and team members to prioritize and deliver high-impact solutions
Requirements
- 3+ years of experience in a software engineering or data engineering role working with large-scale data systems
- Significant experience in Python, Apache Spark, and Delta Lake
- Knowledge of relational data models, SQL, and using them for complex data handling
- Expertise in cloud-native technologies within the Microsoft Azure ecosystem including Databricks and Azure DevOps
- Familiarity with automated testing practices and the ability to design reliable test strategies
- Proficiency in analyzing complex problems, breaking them into actionable steps, and delivering elegant solutions
- Fluency in English and strong communication skills to work effectively within distributed and collaborative teams
Nice to have
- Understanding of Spark Structured Streaming for real-time and near real-time data processing solutions
- Showcase of working on financial data or similar high-complexity data domains
- Competency in identifying and resolving performance bottlenecks in distributed data systems
- Familiarity with documenting data pipeline architecture and best practice implementation approaches
Benefits
- International projects with top brands
- Work with global teams of highly skilled, diverse peers
- Healthcare benefits
- Employee financial programs
- Paid time off and sick leave
- Upskilling, reskilling and certification courses
- Unlimited access to the LinkedIn Learning library and 22,000+ courses
- Global career opportunities
- Volunteer and community involvement opportunities
- EPAM Employee Groups
- Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn