Senior Data Software Engineer (Azure, Spark)
Hybrid in Poland: Gdańsk
Data Software Engineering& 6 others
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a jobWe are looking for a Senior Data Software Engineer to contribute to the design and development of a modern enterprise data platform focused on large-scale financial data processing. The platform is being built to deliver new capabilities rather than maintain legacy systems, with a strong emphasis on data quality, governance, and scalable processing across cloud-native technologies. The role involves working on data ingestion, transformation, modeling, and optimization within a medallion architecture, supporting both structured and complex datasets, including private markets data.
Responsibilities
- Lead the design, development, and optimization of scalable data engineering solutions on Azure, using Spark-based processing (PySpark, SparkSQL, Scala, or equivalent)
- Own end-to-end data transformation pipelines, including ingestion, transformation, storage, and analytics
- Work with Azure-native data services such as Data Factory, Databricks, and Synapse
- Support high-performance data access patterns using Cosmos DB (NoSQL API) where applicable
- Collaborate with data scientists, AI engineers, and product stakeholders to enable data-driven analytics and insights
- Mentor and guide junior engineers, setting coding standards and best practices
- Ensure data quality, security, governance, and performance across platforms
- Contribute to technical decision-making and solution architecture discussions
- Build and maintain data pipelines within a medallion architecture, supporting structured and complex datasets
- Apply CI/CD practices to automate validation and deployment of data pipelines
Requirements
- 5+ years of experience in data engineering, with a focus on large-scale data processing
- Proficiency in Microsoft Fabric, Apache Spark, and Python
- Skills in Pandas, Delta Lake, and Parquet for data processing, storage, and optimization
- Knowledge of Cosmos DB for bitemporal data storage and retrieval
- Familiarity with AWS Glue for supporting ETL workflows
- Expertise in Spark Streaming for near real-time processing with watermarking support
- Competency in CI/CD tools for building automated validation pipelines
- Background in mentoring engineers and establishing coding standards and best practices
- English proficiency at B2 level or higher
Nice to have
- Hands-on experience with Microsoft Fabric and OneLake
- Expertise in Cosmos DB (NoSQL API) specifically
- Familiarity with financial instruments and financial services data
- Familiarity with AI-assisted development tools such as GitHub Copilot and awareness of industry-standard LLMs
- Understanding of Data Science fundamentals and collaboration experience with DS teams
