Lead Data Engineer (PySpark, statistics, MLflow)
Remote in Romania
Data Software Engineering, Azure Databricks
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a jobWe are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models.
Responsibilities
- Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
- Decompose and refactor large legacy notebooks without relying on existing documentation
- Apply statistical techniques to detect outliers and events in sensor time series data
- Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning
- Implement model explainability techniques such as SHAP for existing models
- Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa
- Work independently on data engineering projects with minimal oversight
- Integrate AI coding assistants into daily workflows and critically review their output
- Utilize Window functions to optimize data transformation logic
Requirements
- 5+ years of experience in data engineering with deep knowledge of PySpark and pandas
- Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions
- Capability to read and decompose large legacy notebooks without documentation, and to work independently on them
- Expertise in applied statistics, including outlier detection and event detection in sensor time series
- Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP
- Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa
- Familiarity with using AI coding assistants in daily work and reviewing their output critically
- English proficiency at B2 level or higher
Nice to have
- Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers
- Background in working with sensor or IoT data
- Databricks Machine Learning Professional certification
- German proficiency
