Back to SearchSenior Data Integration Engineer (Azure)
Hybrid in Malaysia
Data Integration& 4 others
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a job
We are looking for a Data Integration Engineer to build and run reliable data pipelines on Microsoft Azure. You will use Azure Databricks, PySpark, Spark SQL, Delta Lake and Azure Data Factory to support reporting and data products using a Medallion Architecture approach. You own delivery, quality and secure operations.
Responsibilities
- Build and maintain data pipelines using Azure Databricks, PySpark and Spark SQL
- Implement Medallion Architecture patterns using Delta Lake tables and workloads
- Develop and schedule orchestration workflows in Azure Data Factory
- Integrate data from Azure Data Lake Storage (ADLS) Gen2, Azure SQL Database and REST application programming interfaces (APIs)
- Implement secret management using Azure Key Vault and Databricks secret scopes
- Maintain continuous integration and continuous delivery (CI/CD) using GitHub Actions and GitHub
- Support reporting enablement and dataset operations in Power BI Service
- Apply security best practices across Azure and Databricks using service principals and role-based access control (RBAC)
Requirements
- Hands-on data engineering experience in production environments
- Strong knowledge of Apache Spark, Spark SQL and Delta Lake
- Experience building pipelines with Medallion Architecture concepts
- Advanced Structured Query Language (SQL) skills, including common table expressions (CTEs), MERGE, window functions and complex transformations
- Python scripting capability for data integration, including REST API calls
- Experience with Azure services such as Azure Data Factory, ADLS Gen2, Azure SQL Database, Azure Synapse Analytics, Azure Key Vault and Azure Identity
- Knowledge of dimensional modeling, including fact tables, dimension tables and star schemas
- Experience implementing data quality checks, logging and monitoring
Nice to have
- Experience with Microsoft Fabric