Intermediate / Senior Data Engineer

Senior AI Engineer

Data Engineer

IAD / SAD / LAD GCP Data Engineer

Manager, Data Science (GNS)

Responsibilities: <ul><li>Designing, implementing, and optimising scalable data pipelines using Python, PySpark, and Databricks.</li><li>Driving the delivery of customer-facing data products through APIs, Databricks-based sharing, and event-driven mechanisms (e.g., Kafka or similar).</li><li>Taking ownership of end-to-end features — from scoping and development to deployment and monitoring.</li><li>Leading and participating in technical design discussions, contributing to architectural improvements and long-term data strategy.</li><li>Mentoring and coaching junior engineers actively while supporting their technical and professional development.</li><li>Collaborating with cross-functional teams to understand business requirements and translate them into technical solutions.</li><li>Conducting peer code reviews, upholding code quality standards, and championing continuous improvement practices.</li><li>Contributing to the team’s agile delivery process and provide input during planning and retrospectives.</li></ul> Requirements: <ul><li>Have extensive experience in data engineering or backend software engineering with a data focus.</li><li>Be proficient in Python, PySpark, and working within Databricks environments.</li><li>Have hands-on experience designing and delivering data products via REST APIs, event-driven systems, or data sharing platforms like Databricks Delta Sharing.</li><li>Have a solid understanding of distributed data processing, ETL/ELT workflows, and data lake/lakehouse architectures.</li><li>Have experience with cloud platforms (e.g., AWS, Azure, GCP) and modern data tooling.</li><li>Have a proven track record of mentoring or coaching other engineers.</li><li>Have effective communication and collaboration skills, including the ability to explain complex concepts to non-technical stakeholders.</li><li>Be familiar with agile development practices</li></ul>

Role Description: <ul><li>We are building a new data engineering team to support a strategic move from legacy data estates to a modern data platform built on Databricks.</li><li>This role plays a key part in re-engineering existing systems, improving scalability, and enabling future innovation.</li></ul> Responsibilities: <ul><li>Transition legacy data systems to Databricks.</li><li>Build and maintain robust, scalable data pipelines.</li><li>Work closely with subject matter experts, product teams, and stakeholders to understand business logic.</li><li>Apply best practices in data engineering and programming.</li><li>Mentor junior engineers and contribute to knowledge sharing.</li><li>Participate in code reviews and technical discussions.</li></ul> Requirements: <ul><li>Hands-on experience in software or data engineering roles.</li><li>Strong proficiency in Python and working knowledge of PySpark.</li><li>Experience or strong interest in Databricks.</li><li>Solid understanding of SQL and structured data.</li><li>Experience with cloud platforms (AWS preferred; Azure/GCP acceptable).</li><li>Familiarity with REST APIs and event-driven architectures.</li><li>Experience using Git or similar version control systems.</li><li>Strong problem-solving skills and collaborative mindset.</li></ul> Preferred Experience <ul><li>Experience working on data platform modernization or migration projects.</li><li>Ability to refactor and improve legacy codebases.</li><li>Exposure to AI/ML data pipelines is an advantage.</li></ul> Why Join Us <ul><li>Opportunity to work on a strategic, high-impact data platform transformation.</li><li>Exposure to cutting-edge technologies including Databricks and cloud platforms.</li><li>Collaborative, inclusive team culture with strong mentorship.</li><li>Clear opportunities for career growth and skill development.</li></ul>

Role Overview: <ul><li>We are building a new data engineering team to support a strategic move from legacy data estates to a modern data platform built on Databricks.&nbsp;</li><li>This role plays a key part in re-engineering existing systems, improving scalability, and enabling future innovation.</li></ul> Responsibilities: <ul><li>Lead the transition of legacy data systems to Databricks, ensuring reliability and performance.</li><li>Design, build, and maintain robust, scalable data pipelines to support analytical and operational use cases.</li><li>Collaborate closely with subject matter experts, product teams, and business stakeholders to understand and implement business logic accurately.</li><li>Apply data engineering and programming best practices to ensure high-quality, maintainable, and secure solutions.</li><li>Mentor junior engineers, fostering technical growth and promoting knowledge sharing across the team.</li><li>Participate in code reviews, architectural discussions, and technical decision-making to uphold engineering standards.</li></ul> Requirements:&nbsp; Required Skills &amp; Experience: <ul><li>Hands-on experience in software or data engineering roles.</li><li>Strong proficiency in Python and working knowledge of PySpark.</li><li>Experience or strong interest in Databricks.</li><li>Solid understanding of SQL and structured data.</li><li>Experience with cloud platforms (AWS preferred; Azure/GCP acceptable).</li><li>Familiarity with REST APIs and event-driven architectures.</li><li>Experience using Git or similar version control systems.</li><li>Strong problem-solving skills and collaborative mindset.</li></ul> Preferred Experience: <ul><li>Experience working on data platform modernization or migration projects.</li><li>Ability to refactor and improve legacy codebases.</li><li>Exposure to AI/ML data pipelines is an advantage.</li></ul>

Responsibilities: Architecture &amp; Design:&nbsp; <ul><li>Design end-to-end data architectures that integrate real-time streaming (Kafka, GCP Pub/sub) and batch processing (Big Query OR Databricks).</li></ul> Pipeline Development:&nbsp; <ul><li>Build and maintain robust ETL/ELT pipelines flowing into BigQuery/Databricks, ensuring data is clean, modeled, and ready for consumption.</li></ul> Custom Integration:&nbsp; <ul><li>Develop custom APIs and microservices using Java/Python/C# to facilitate data movement between niche systems and the data lakehouse.</li></ul> Infrastructure Management:&nbsp; <ul><li>Deploy and manage data workloads on Google Kubernetes Engine (GKE), ensuring high availability and cost-efficiency.</li></ul> Technical Leadership:&nbsp; <ul><li>Mentor junior engineers, conduct code reviews, and establish DataOps best practices (CI/CD, automated testing, and monitoring).</li></ul> Data Reliability:&nbsp; <ul><li>Implement data quality frameworks to ensure accuracy and lineage across the hybrid environment.</li></ul> Requirements: <ul><li>Experience with Terraform &amp; Azure Dev Ops (ADO).</li><li>Knowledge of workflow orchestrators like Apache Airflow (Cloud Composer).</li><li>Experience implementing Data Governance (metadata management, data cataloging).</li><li>Data Processing &amp; Storage</li><li>BigQuery: Expert-level SQL and experience with partition/cluster optimization.</li><li>Databricks: Deep knowledge of the Lakehouse architecture, Delta Lake, and Spark (PySpark/Scala) Software Engineering &amp; APIs</li><li>Languages: Proficiency in Java/Python/C# for backend development and custom data connectors.</li><li>API Design: Proven ability to build and consume RESTful APIs for data ingestion Streaming &amp; Infrastructure</li><li>Kafka/GCP Pub-Sub: Experience building event-driven architectures and managing producers/consumers.</li><li>GKE: Proficiency in containerization (Docker) and orchestrating data workloads on Kubernetes.</li></ul>

Responsibilities: <ul><li>Manage a team of data scientists and engineers to design, develop, and deploy advanced AI solutions.</li><li>Lead the development of AI and machine learning models to solve business problems.</li><li>Research, build, and deploy models based on shallow and deep machine learning.</li><li>Train robust NLP-based models on large datasets.</li><li>Collaborate with cross-functional teams to align AI solutions with business goals and customer needs.</li><li>Design APIs for disseminating data science outputs.</li><li>Oversee the scaling of production systems to meet growing demands.</li><li>Working closely with other data scientists and engineers to design, develop, and deploy AI solutions.</li><li>Analyzing large datasets to identify patterns and trends that can inform AI development.</li><li>Communicating findings and insights to stakeholders in a clear and concise manner.</li><li>Staying current with the latest AI and machine learning trends and technologies.</li><li>Mentoring junior data scientists and provide guidance on AI and machine learning best practices.</li><li>Driving best practices across our data science functions.</li><li>Setting strategy and team direction, with a clear vision for Data Science.</li><li>Working with product leaders to apply data science solutions.</li><li>Evaluating and creating new frameworks and defining methodologies and governance.</li><li>Leading small- to medium-sized teams (direct or indirect).</li><li>Setting a high-bar for model development.</li></ul> Requirements: <ul><li>Have substantial experience in a senior data scientist role.</li><li>Have experience in leading complex projects and customer relationships.</li><li>Have significant experience in managing larger Technical teams (direct or indirect).</li><li>Have experience in mentoring or training others, acting as a subject matter expert to guide colleagues.</li><li>Be a natural problem solver, able to take a lead in collaborating to resolve issues.</li><li>Have excellent communication skills and the ability to convey information clearly to senior stakeholders.</li></ul> Highly Desired Skills and Experience: <ul><li>PhD or Masters degree in Computer Science, Mathematics, Statistics, or a related field.</li><li>7-10+ years of experience in AI and machine learning.</li><li>Deep understanding of machine learning algorithms, including deep learning, gradient boosting, and random forests.</li><li>Experience working directly with large language models and Transformer based architectures including BERT, RoBERTa, T5 etc.</li><li>Working knowledge of applying recent LLMs including ChatGPT, GPT 3.5, OPT, BLOOM, etc.</li><li>Experience with reinforcement learning, prompt engineering, hallucination mitigation.</li><li>Working understanding of the business risks associated with applying LLM in a business.</li><li>Experience working with large datasets and distributed computing systems (e.g., Hadoop, Spark).</li><li>Strong coding skills in Python or another programming language.</li><li>Excellent communicator, with the ability to translate complex technical concepts to non-technical stakeholders.</li><li>Strong problem-solving skills and the ability to think outside the box. </li></ul>

Skills

Industry

Location

Jobtitle

5 pyspark Jobs found

Intermediate / Senior Data Engineer

Senior AI Engineer

Data Engineer

IAD / SAD / LAD GCP Data Engineer

Manager, Data Science (GNS)

WhiteCrow
WhiteCrow is on a mission to connect companies with the best talent across the globe.

EMPLOYER

TALENT

COMPANY

Contact Us

JOBS BY SKILLS

JOBS BY FUNCTION

JOBS BY INDUSTRY

JOBS BY LOCATIONS

MODEL JOBS

JOBS BY CITY

Skills

Industry

Location

Jobtitle

5 pyspark Jobs found

Intermediate / Senior Data Engineer

Senior AI Engineer

Data Engineer

IAD / SAD / LAD GCP Data Engineer

Manager, Data Science (GNS)

WhiteCrow WhiteCrow is on a mission to connect companies with the best talent across the globe.

EMPLOYER

TALENT

COMPANY

Contact Us

JOBS BY SKILLS

JOBS BY FUNCTION

JOBS BY INDUSTRY

JOBS BY LOCATIONS

MODEL JOBS

JOBS BY CITY

WhiteCrow
WhiteCrow is on a mission to connect companies with the best talent across the globe.