Back to SearchSenior Big Data Engineer
Remote in Portugal
Data Software Engineering
& 5 others
Choose an option
We are seeking a highly skilled and experienced Senior Big Data Engineer to join an exciting project focused on rearchitecting an on-premises Ruby application, ETL pipelines, and underlying data storage and reports to a cloud-native AWS environment.
Our client is a world-renowned technology company from Silicon Valley, focused on revolutionizing data systems. You will work with world-class experts and technologies in the heart of Silicon Valley, developing products and services that billions of people use every day.
Responsibilities
- Design and implement scalable ETL pipelines in a cloud-native AWS environment
- Develop and maintain data processing workflows using Python, Spark, and Airflow
- Optimize data storage solutions and implement advanced data management strategies using AWS and Redshift
- Ensure the platform meets non-functional requirements by defining robust testing strategies and execution plans
- Collaborate with cross-functional teams to create and maintain decision and change logs, ensuring consistent communication and tracking of technical decisions
- Proactively identify opportunities to enhance the quality of the product while managing scope effectively
- Monitor and fine-tune data pipelines for performance, scalability, and maintainability
- Provide technical guidance and mentorship to team members, promoting best practices in data engineering
- Troubleshoot and resolve complex issues in data workflows and pipelines
Requirements
- 3+ years of experience working in Big Data engineering roles
- Strong data engineering and overall engineering skills, including expertise in testing and quality assurance
- Proficiency in Python
- Background in Spark, Airflow, and AWS technologies like Redshift
- Proven experience in decision-making processes with the ability to document and maintain decision and change logs
- Understanding of strategies to meet non-functional requirements such as performance, scalability, and testing
- Capability to design solutions that enhance product quality without significantly increasing scope
- Excellent command of English (B2+ level), both written and spoken
Nice to have
- Knowledge of Ruby, Iceberg, Trino, Cassandra, and Tableau