Senior Data Engineer, Azure Data Factory

Remote in Georgia,

Data Integration

Facebook LinkedIn Send via email

Looking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a job

Location-specific conditions & benefits*

Choose an option

We are seeking a skilled Senior Data Engineer with deep expertise in Azure Data Factory to design, build and optimize robust data pipelines across analytical workloads. In this role, you will author and deploy ADF pipelines, implement incremental load patterns and tune database performance to support enterprise-grade BI solutions.

Responsibilities

Author ADF pipelines using Copy Activity, Script Activity, ForEach, Execute Pipeline as well as parameterized datasets and linked services
Debug failed runs in the ADF monitoring view using Azure IR
Connect ADF instances to a Git repository such as Azure Repos or GitHub, working with the collaboration branch and feature branch model
Publish ADF artifacts from the Git branch and deploy them to staging and production instances through a CI/CD pipeline
Implement SQL Server Change Tracking with CHANGETABLE queries, version-based watermarking and the standard incremental load pattern documented by Microsoft
Design star schemas with fact/dimension modeling for analytical workloads, including choosing grain, identifying dimensions and handling slowly changing dimensions
Build merge functions and bulk loading processes into delta tables using COPY protocol and PL/pgSQL
Optimize PostgreSQL performance through checkpoint config, WAL tuning and query optimization
Configure PgBouncer session and transaction mode while understanding the I/O constraint chain across VM ceiling, disk throughput and connection limits
Define indexing strategies for analytical queries and debug performance with EXPLAIN ANALYZE

Requirements

3+ years of experience in data engineering with proven hands-on ADF pipeline authoring and deployment
Expertise in ADF's Git-connected authoring model, export parameterization and deployment of artifacts through a CI/CD pipeline to multiple environments
Proficiency in SQL Server Change Tracking, CHANGETABLE queries and version-based watermarking
Skills in PostgreSQL fundamentals including COPY protocol, PL/pgSQL and indexing strategies for analytical queries
Competency in PostgreSQL performance tuning covering checkpoint config, WAL tuning and PgBouncer session vs. transaction mode
Background in star schema design with fact/dimension modeling and slowly changing dimensions
Understanding of how BI tools generate SQL against star schemas
Familiarity with EXPLAIN ANALYZE for query performance debugging
English proficiency at B2 level or higher

Nice to have

Showcase of ThoughtSpot-specific experience
Familiarity with live-query BI tools such as Looker, Tableau live connections or Power BI DirectQuery