Back to SearchSenior AI Engineer with Databricks
Remote in Kazakhstan, Georgia
AI Engineering
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a job
Choose an option
We are looking for a Senior AI Engineer with Databricks expertise to design, deploy and maintain scalable machine learning pipelines using the Databricks platform. In this role, you will deliver production-ready ML pipelines, automated training and retraining workflows, deployed models, monitoring dashboards and CI/CD pipelines for ML systems.
Responsibilities
- Design, implement and maintain end-to-end ML pipelines on Databricks
- Build workflows for data ingestion, preprocessing, feature engineering, training and inference
- Leverage PySpark, Spark ML and Databricks notebooks/jobs
- Manage model versioning, experiment tracking and reproducibility using MLflow
- Package and deploy models for batch and real-time inference
- Monitor model performance, drift and retraining cycles
- Develop scalable ETL/ELT pipelines using Databricks Delta Lake
- Optimize data storage and access patterns through partitioning, Z-ordering and caching
- Integrate with data sources such as Azure Data Lake, S3, APIs and databases
- Implement CI/CD pipelines for ML workflows using Azure DevOps, GitHub Actions and Databricks Repos and Jobs API
- Configure clusters, autoscaling and cost optimization while applying Infrastructure as Code with Terraform, ARM and Bicep
- Implement logging, alerting and observability to ensure high availability and fault tolerance of ML systems
Requirements
- 3+ years of experience in machine learning engineering or related roles
- Expertise in the Databricks platform including workspaces, jobs and clusters
- Proficiency in Apache Spark, PySpark and Python with pandas and scikit-learn
- Skills in MLflow for tracking, registry and deployment
- Competency in CI/CD pipelines, Docker containerization and REST APIs for model serving
- Familiarity with version control using Git
- Background in Azure including Azure Databricks, ADLS, ACR and AML
- Knowledge of data preprocessing, feature engineering and model training and evaluation
- Understanding of libraries such as XGBoost, LightGBM and CatBoost
- English proficiency at B2 level or higher
Nice to have
- Familiarity with AWS including S3, EMR and SageMaker
- Skills in streaming pipelines with Spark Structured Streaming and Databricks Feature Store
- Knowledge of Kubernetes
- Competency in monitoring tools such as Prometheus and Grafana
- Experience with large-scale production systems