Apache Spark Jobs

Viewing 1-10 out of 131 jobs found

Sort By

relevant

hot

Senior Data Software Engineer (Databricks)

Hybrid in Spain: Málaga & 1 others

Data Software Engineering

We are seeking a Senior Data Software Engineer skilled in Databricks to join our team. You will be integral to our efforts, bringing a deep understanding of data engineering, expertise in Databricks, and an open-minded approach to our collaborative and friendly work environment. This role is ideal for a professional eager to develop scalable data solutions and drive innovation in our data practices. This position offers a hybrid work setup with flexibility to work from Malaga or Madrid, with occasional office visits required.

responsibilities

Design, develop, and maintain scalable data pipelines and architectures
Optimize data models and enhance ETL processes utilizing Databricks and related technologies
Implement rigorous data quality checks and monitoring to guarantee high integrity
Keep abreast of the latest trends and technologies in data engineering and advocate for the adoption of beneficial new tools
Troubleshoot and resolve data-related issues promptly
Participate in code reviews ensuring adherence to high standards of code quality

requirements

Hands-on experience with Databricks, including Delta Lake, workflows, Delta Live Tables, along with deployment and versioning strategies
Solid understanding of data architectures and data modeling skills
Expertise in Spark, using either Scala or PySpark
Proficiency in cloud-native technologies and software engineering best practices such as containers, unit tests, linting, and code style checks
Engineering background in AWS, Azure, or GCP
Experience with big data and performance optimization of data-intensive applications
Proactive approach and prior client-facing experience
Ability to navigate ambiguity and work independently
Desire to engage in a transparent, fast-moving startup environment
Fluent English communication skills at a B2+ level or higher
Solid communication skills in Spanish (at least C1)

Senior Data Software Engineer [Scala]

Remote in Ukraine

Data Software Engineering

We are seeking a Senior Data Software Engineer to lead the migration of Ad Tech workflows from Spark on EKS and Apache Flink to Databricks on AWS. This strategic role will focus on creating scalable, high-performance data pipelines, driving optimization efforts, and partnering with cross-functional teams to integrate solutions into the Unity Catalog.

responsibilities

Migrate client workflows from Spark on EKS and Flink to Databricks on AWS while ensuring reliability and performance
Align solutions with business needs and architectural standards by partnering with Data Architects and Solution Leads
Review code, enforce best practices, and maintain coding standards across projects
Partner with DevOps and QA teams to guarantee smooth deployment and validate data flows
Enable integration of data assets into Unity Catalog, including defining schema and aligning access controls
Monitor, troubleshoot, and resolve production issues to maintain pipeline reliability and performance

requirements

5+ years of experience in Scala programming, ideally with a Java background
Experience managing data workflows with tools such as Airflow and modern orchestration systems
Strong expertise in Apache Spark (Scala and PySpark), with competency in Python as a plus
Knowledge of data modeling, schema design, performance tuning, and SQL
Proven understanding of version control (Git) and Agile development practices
Strong collaboration skills to work across global, cross-functional teams
Willingness to adjust working hours partially to support USA West Coast time zone

nice to have

Background in Apache Flink, especially in migration strategies
Understanding of Unity Catalog and catalog integration (e.g., AWS Glue, Hive)
Skills in data quality principles and test automation for data pipelines
Familiarity with large-scale workflow orchestration tools like Airflow

Lead Data Software Engineer

Remote in Mexico & 4 others

Data Software Engineering

We are seeking an experienced Lead Data Software Engineer for a remote position to support our team’s success and advance our Data Science initiatives. In this role, you will play a pivotal part in designing and maintaining datamarts while ensuring scalable and efficient data solutions tailored to meet client needs. You will focus on developing and managing data pipelines, providing on-call support, and working closely with Data Science teams to deliver high-quality results. Additionally, you will lead a team of skilled professionals, offering mentorship and guidance to drive innovation and excellence within the industry.

responsibilities

Lead and mentor the Data Software Engineering team, fostering a culture of continuous improvement and professional growth
Collaborate across disciplines to deliver high-quality data solutions aligned with project goals and deadlines
Establish and maintain effective Data Software Engineering processes, emphasizing automation, efficiency, and innovation
Implement and sustain scalable data solutions using AWS services
Oversee optimization of workflows and data processing through Apache Airflow and Apache Spark
Continuously evaluate industry trends and best practices to refine and adopt advanced Data Software Engineering methodologies
Provide on-call support for data pipelines and datamarts to ensure operational efficiency
Direct the creation and deployment of REST APIs to enable seamless data integration and communication
Engage with clients to understand their needs and deliver tailored solutions that meet their requirements
Manage team structure and organization to ensure timely and efficient delivery of projects
Work with stakeholders, demonstrating strong communication and leadership capabilities

requirements

At least 5 years of experience as a Data Software Engineer, working on complex projects and large-scale data infrastructures
Minimum of 1 year of leadership experience, successfully managing and motivating a team of Data Software Engineers
Expertise in Amazon Web Services (AWS), including designing and implementing scalable data solutions
Extensive knowledge of Apache Airflow and Apache Spark for optimizing workflows and data processing
Advanced proficiency with one or more CI/CD tools, such as Jenkins, to streamline data pipeline delivery
Strong skills in Python and SQL for building and maintaining data pipelines and ETL processes
Experience with Databricks and PySpark for data analysis and processing
Familiarity with REST APIs for efficient data integration and communication
Excellent analytical skills for problem-solving and decision-making in complex environments
Strong client-facing abilities, ensuring effective collaboration and clear communication to achieve project objectives
Exceptional organizational and team management skills for delivering projects efficiently
Upper-intermediate English proficiency for effective communication, presentations, and discussions with stakeholders

nice to have

Experience with Redshift for data warehousing and analysis

Senior Data Software Engineer

Remote in Colombia

Data Software Engineering

We are seeking a highly motivated Senior Data Software Engineer to join our remote team, working with the world's leading source of news and information for professional markets. As a Senior Data Software Engineer, you will be responsible for developing and building Spark applications, working with large datasets and Cloud technologies. If you have a solid understanding of the core AWS building blocks, including workloads, stacks, instances, containers, and basic security knowledge - we invite you to apply!

responsibilities

Develop and build Spark applications, working with large datasets
Perform data parsing and manipulation using Scala and Python
Collaborate with cross-functional teams to understand product requirements and user needs
Participate in design reviews, offering input on requirements, product design, and potential challenges
Contribute to the preparation of automated test cases using tools like CodeBuild/CodeDeploy or other preferred automation frameworks
Prepare detailed reports covering all aspects of software development, including identified issues and development outcomes

requirements

A minimum of 3 years of experience in Data Software Engineering, demonstrating your proficiency in developing and building Spark applications
Solid understanding of the core AWS building blocks, including workloads, stacks, instances, containers, VPCs, networking, internetworking, routing, core services like DNS
Proficiency in Scala and Python, enabling you to work with the Scala ecosystem and perform ETL/ELT jobs
Experience working with Apache Hadoop and Databricks
Strong focus on teamwork and exceptional interpersonal skills, allowing you to collaborate effectively with cross-functional teams
Fluent spoken and written English at an Upper-Intermediate level or higher

nice to have

Work experience with the following technologies: S3, Oracle, Elastic Search, Step Functions, Dynamo, CloudTrail/CloudWatch, DataDog, SNS/SQS, or Athena

Senior Data Software Engineer

Remote in Colombia

Data Software Engineering

Join our remote team as a Senior Data Software Engineer and work with a Swiss-American pharmaceutical and medical device company specializing in eye care products. With a focus on vision care and surgical products, this company is committed to improving eye health worldwide. As a Senior Data Software Engineer, you will be responsible for designing, developing, and maintaining data pipelines using Apache Spark and PySpark. You will collaborate with cross-functional teams to understand business requirements and translate them into technical solutions, ensuring data security and compliance with data governance policies.

responsibilities

Develop and maintain data pipelines and analytics solutions
Optimize Spark jobs for performance and scalability, leveraging AWS services
Contribute to the development of data architecture and data modeling
Ensure data accuracy and quality control through unit tests and columnar-based file types
Participate in code reviews and contribute to the development of best practices for data engineering
Collaborate with cross-functional teams to understand business requirements and data needs, contributing to the development of data-driven solutions
Stay up-to-date with the latest data engineering technologies and trends

requirements

A minimum of 3 years of experience in Data Software Engineering, with a strong understanding of the Data domain
Experience with Apache Spark and PySpark, as well as SQL and Big Data technologies
Proficiency in Amazon Web Services (AWS), including AWS CLI, S3, Glue, Lambda, Athena, and Step Functions
Familiarity with columnar-based file types and unit testing
Excellent communication skills and strong critical thinking capabilities to effectively convey feedback and insights
Fluent spoken and written English at an Upper-Intermediate level or higher, enabling effective communication

nice to have

Experience with CI/CD tools such as Atlassian Bitbucket, AWS CodeBuild, and Terraform

Senior Data Quality Engineer

Remote in Colombia & 4 others

Data Quality Engineering

We are seeking an experienced and accomplished Senior Data Quality Engineer to join our team, driving the reliability, accuracy, and efficiency of our data systems and processes at scale. If you are passionate about shaping high-impact data quality initiatives and are eager to work with advanced technologies, this role will empower you to influence the future of our data landscape.

responsibilities

Oversee end-to-end data quality strategy, ensuring rigorous testing and reliability of data products and processes
Drive data quality initiatives while instilling best practices across multiple teams and projects
Define and enforce advanced testing methodologies and frameworks to ensure enterprise-level data quality
Prioritize and manage complex data quality tasks, optimizing efficiency under tight timelines and competing demands
Architect and maintain robust testing strategies tailored to evolving system architectures and data pipelines
Advise on resource allocation, setting priorities for testing aligned with regulatory and business standards
Establish and continuously enhance a robust data quality governance framework, overseeing compliance with industry standards
Develop, scale, and refine automated data quality validation pipelines for production systems
Collaborate at a high level with cross-functional teams to resolve infrastructure challenges and optimize performance
Mentor junior engineers while maintaining comprehensive documentation, including versions of test strategies and advanced test plans

requirements

3+ years of professional experience in Data Quality Engineering
Advanced programming skills in Python
Deep expertise in Big Data platforms, including Hadoop ecosystem tools (HDFS, Hive, Spark) and modern streaming platforms (Kafka, Flume, Kinesis)
Strong practical knowledge of NoSQL databases such as Cassandra, MongoDB, or HBase, with a track record of handling enterprise-scale datasets
Advanced skills in data visualization and analytics tools (e.g., Tableau, Power BI, Tibco Spotfire) to support decision-making
Extensive hands-on experience with cloud ecosystems like AWS, Azure, and GCP, including a strong understanding of complex multi-cloud architectures
Demonstrated expertise with relational databases and SQL (PostgreSQL, MSSQL, MySQL, Oracle) in high-volume, real-time environments
Proven ability to implement, troubleshoot, and scale ETL processes using tools like Talend, Informatica, or similar platforms
Experience deploying and integrating MDM tools into existing workflows, with knowledge of performance testing tools such as JMeter
Advanced experience in version control systems (Git, GitLab, or SVN) and automation/scripting for large-scale systems
Comprehensive knowledge of modern testing frameworks (TDD, DDT, BDT) and their application in data-focused environments
Familiarity with CI/CD practices, including implementation of pipelines using tools like Jenkins or GitHub Actions
Highly developed problem-solving abilities and an analytical mindset capable of interpreting complex datasets into actionable business outcomes
Exceptional verbal and written communication skills in English (B2 level or higher), paired with experience guiding discussions with stakeholders

nice to have

Extensive hands-on experience with programming languages like Java, Scala, or advanced Bash scripting for production-level data solutions
Advanced knowledge of XPath and its applications in data validation or transformation workflows
Experience designing customized data generation tools and sophisticated synthetic data techniques for testing scenarios

Senior Data Software Engineer

Remote in Colombia & 4 others

Data Software Engineering

We are looking for a remote Senior Data Software Engineer to join our team. This role requires an experienced professional with strong expertise in Apache Spark, Microsoft Azure, and Python. You will be contributing to a high-profile project for a globally recognized brand, developing advanced data solutions that drive business success and innovation.

responsibilities

Collaborate with cross-functional teams to design, develop, and maintain data integration solutions
Create and optimize data pipelines using Apache Spark, Microsoft Azure, and Python
Ensure data pipelines are robust, scalable, and easy to maintain
Transform data science models into production-ready systems
Build and manage forecasting models to support critical business decisions
Ensure consistent data quality and integrity across all sources
Monitor and refine data pipelines to improve performance and efficiency
Partner with data scientists to design and deploy machine learning models
Continuously enhance data integration solutions to meet evolving business needs
Stay updated on the latest technologies and trends in data software engineering

requirements

At least 3 years of experience in data software engineering or a similar role
Expertise in Apache Spark, Microsoft Azure, and Python
Strong knowledge of forecasting models, MLOps practices, and data science principles
Hands-on experience with Databricks for creating and optimizing data pipelines
Proven ability to operationalize data science models for production environments
Proficiency with Git for version control and collaboration
Solid understanding of Azure concepts such as clouds, regions, ADLS, and compute services
Exceptional analytical and problem-solving skills with a creative approach to technical challenges
Experience working in Agile development environments
Advanced English communication skills, both written and verbal, at a B2+ level

nice to have

Experience with Panda for data manipulation and analysis
Advanced knowledge of SQL and relational database systems
Understanding of statistical models and experience developing them with Python or Spark
Familiarity with tools and technologies related to machine learning

Senior Data Delivery Manager

Remote in Colombia & 4 others

Data Delivery Management

We are looking for a highly skilled and motivated Senior Data Delivery Manager to lead the delivery of data projects, ensuring technical quality and effective collaboration with stakeholders. This role requires overseeing delivery processes, resource planning, and client engagement, with a focus on Big Data solutions and cloud-based architectures.

responsibilities

Manage end-to-end delivery for assigned accounts and serve as the primary contact for delivery-related issues
Ensure compliance with delivery processes and governance frameworks in line with chosen models
Assign staffing and resources to align with delivery plans, ensuring appropriate expertise and experience
Define technical solution architectures that meet quality standards and align with future requirements
Coordinate across stakeholders and disciplines to achieve project objectives
Foster and maintain strong client relationships while addressing potential risks
Prepare customer proposals, statements of work, and other delivery-related documentation
Uphold best practices, methodologies, and agile models throughout the delivery lifecycle
Develop risk mitigation strategies and integrate them into the delivery process
Track, evaluate, and improve delivery productivity to drive continuous enhancements
Provide delivery expertise as part of sales efforts and bid processes
Offer mentorship and guidance to other delivery managers in the team

requirements

9+ years of experience in Data and Analytics project delivery, involving Big Data, Data Warehousing, and Business Intelligence
Background in Python, Scala, or Java with familiarity with Big Data tools like Hadoop, Spark, Hive, and Kafka
Expertise in data platform engineering solutions for batch and streaming pipelines
Proficiency in SQL with experience in building and maintaining Big Data solutions
Experience leading on-prem to cloud migration projects with platforms like AWS, Azure, or GCP
Solid understanding of data architecture principles and approaches to address data security challenges
Knowledge of data visualization techniques and tools, along with familiarity with classical analytics concepts
Capability to manage multiple priorities in a fast-paced environment with strong client-facing and communication skills (C1 English level)
Understanding of project delivery frameworks, including agile methodologies and change management

Senior Data Software Engineer

Remote in Colombia & 4 others

Data Software Engineering

We are seeking a remote Senior Data Software Engineer to join our team. The right candidate will have a strong technical background with expertise in Apache Spark, Microsoft Azure, and Python. This position offers a unique opportunity to work on a high-impact project with one of the world's most recognized brands.

responsibilities

Collaborate with cross-functional teams to build and maintain the data integration solution
Develop, build, and optimize data pipelines using Apache Spark, Microsoft Azure, and Python
Ensure data pipelines are scalable, maintainable, and reliable
Take data science models and make them production ready
Develop and maintain forecasting models to support business decisions
Ensure data quality and consistency across all data sources
Monitor and optimize data pipelines to ensure efficient and effective data processing
Collaborate with data scientists to develop and implement machine learning models
Work with the team to continuously improve and optimize the data integration solution for Fedex
Stay current with emerging technologies and trends in data software engineering

requirements

At least 3+ years of experience in data software engineering or similar roles
Expertise in Apache Spark, Microsoft Azure, and Python
Strong knowledge of forecasting models, data science, and MLOps
Experience working with Databricks to build, maintain, and optimize data pipelines
Ability to take data science models and make them production ready
Experience with Git for version control
Understanding of basic Azure concepts including clouds, regions, ADLS, and compute
Strong analytical and problem-solving skills with the ability to think critically and creatively
Experience working in Agile development environments
Excellent English communication skills, both written and verbal (B2+ level)

nice to have

Experience with Panda for data manipulation and analysis
Strong knowledge of SQL and relational tables
Understanding of statistical models and the ability to develop models utilizing Python, Spark, etc.
Knowledge of ML tools space

Senior Data Software Engineer

Hybrid in Serbia

Data Software Engineering

We are looking for a Senior Data Software Engineer. If you are looking to give your career a real boost with a global leader in digital transformation, EPAM is the perfect choice. If you’d like to learn more about this position and project, apply now.

responsibilities

Design, construct, install, test and maintain highly scalable and optimized data pipelines with state-of-the-art monitoring and logging practices
Bring together large, complex, and sparse data sets to meet functional and non-functional business requirements and use a variety of languages, tools and frameworks to marry data
Design and implement data tools for analytics and data scientist team members to help them build, optimize, and tune use cases
Tackle challenging and varied problems related to gaining further maturity in Data Platform tools, processes, and engineering capabilities
Focus on engineering excellence to deliver high-quality solutions that provide leverage for the company's objectives

requirements

3+ years of hands-on experience with data processing-focused projects
Proficiency in Java/Python/Scala and SQL
Knowledge of Apache Spark
Experience with one of the major cloud providers: AWS, Azure, or GCP
Hands-on experience with selected data processing technologies, e.g., Hadoop, MongoDB, Cassandra, Kafka, Elasticsearch, Python libraries (Pandas/NumPy/etc.), and data processing tools of cloud providers (EMR, Glue, Data Factory, Big Table, etc.)
Relevant experience with version control and code review
Understanding of Agile methodologies
Basic knowledge of Linux and Bash scripting
Fluency in English, both written and spoken, at a minimum B2 level

nice to have

Good hands-on experience with Databricks and Delta Lake
Ability to build Apache Airflow pipelines
Experience with the Snowflake platform

Let us find a perfect job for you

Share your CV and pass our review to get a personalized job offer even if you didn't find a job on the site.

find me a job

Positions covered by Labor Condition Applications pursuant U.S. law can be found here.

Viewing 1-10 out of 131 jobs found

Apache Spark Jobs

FILTERS

Workplace type

Open to relocation

Specializations

Skills

Seniority