IAD / SAD / LAD GCP Data Engineer

Our client is an American multinational shipping & receiving and supply chain management company founded in 1907. Operating in more than 220 countries and territories, they are committed to moving the world forward by delivering what matters. Beginning as a small messenger service and a $100 loan, they now have more than 500,000 users around the globe. They are a transportation and logistics leader, offering innovative solutions to customers, big and small. They understand and appreciate their responsibility to help build safer, stronger and more resilient communities founded on justice and economic opportunity for all, supported by a healthy, sustainable global environment.

Responsibilities:

Architecture & Design: 

• Design end-to-end data architectures that integrate real-time streaming (Kafka, GCP Pub/sub) and batch processing (Big Query OR Databricks).

Pipeline Development: 

• Build and maintain robust ETL/ELT pipelines flowing into BigQuery/Databricks, ensuring data is clean, modeled, and ready for consumption.

Custom Integration: 

• Develop custom APIs and microservices using Java/Python/C# to facilitate data movement between niche systems and the data lakehouse.

Infrastructure Management: 

• Deploy and manage data workloads on Google Kubernetes Engine (GKE), ensuring high availability and cost-efficiency.

Technical Leadership: 

• Mentor junior engineers, conduct code reviews, and establish DataOps best practices (CI/CD, automated testing, and monitoring).

Data Reliability: 

• Implement data quality frameworks to ensure accuracy and lineage across the hybrid environment.

Requirements:

• Experience with Terraform & Azure Dev Ops (ADO).
• Knowledge of workflow orchestrators like Apache Airflow (Cloud Composer).
• Experience implementing Data Governance (metadata management, data cataloging).
• Data Processing & Storage
• BigQuery: Expert-level SQL and experience with partition/cluster optimization.
• Databricks: Deep knowledge of the Lakehouse architecture, Delta Lake, and Spark (PySpark/Scala) Software Engineering & APIs
• Languages: Proficiency in Java/Python/C# for backend development and custom data connectors.
• API Design: Proven ability to build and consume RESTful APIs for data ingestion Streaming & Infrastructure
• Kafka/GCP Pub-Sub: Experience building event-driven architectures and managing producers/consumers.
• GKE: Proficiency in containerization (Docker) and orchestrating data workloads on Kubernetes.