Skip To Main Content
backBack to Search

Lead Data Engineer (PySpark, statistics, MLflow)

Remote in Romania
Data Software Engineering, Azure Databricks
Looking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a job

We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models.

Responsibilities
  • Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
  • Decompose and refactor large legacy notebooks without relying on existing documentation
  • Apply statistical techniques to detect outliers and events in sensor time series data
  • Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning
  • Implement model explainability techniques such as SHAP for existing models
  • Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa
  • Work independently on data engineering projects with minimal oversight
  • Integrate AI coding assistants into daily workflows and critically review their output
  • Utilize Window functions to optimize data transformation logic
Requirements
  • 5+ years of experience in data engineering with deep knowledge of PySpark and pandas
  • Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions
  • Capability to read and decompose large legacy notebooks without documentation, and to work independently on them
  • Expertise in applied statistics, including outlier detection and event detection in sensor time series
  • Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP
  • Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa
  • Familiarity with using AI coding assistants in daily work and reviewing their output critically
  • English proficiency at B2 level or higher
Nice to have
  • Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers
  • Background in working with sensor or IoT data
  • Databricks Machine Learning Professional certification
  • German proficiency