Data Engineer Lead - Python and PySpark

Responsibilities:

Data Ingestion and Extraction:

• Develop and optimize data ingestion pipelines to efficiently extract data from various sources e.g. databases, APIs, files etc.
• Ensure data quality and integrity through robust validation and cleansing processes

Data Transformation and Processing:

• Design and implement data transformation logic using Python and PySpark to convert raw data into a suitable format for analysis
• Optimise data processing pipelines for performance and scalability

Data Storage and Management:

• Select and implement appropriate data storage solutions (cloud-based data warehouses, data lakes, etc.) based on data volume and complexity
• Develop strategies for data retention, backup, and recovery

ETL/ELT Development:

• Build and maintain efficient ETL/ELT processes to move data between different systems and platforms
• Optimize ETL/ELT pipelines for performance and cost-effectiveness

Cloud Infrastructure:

• Leverage cloud-based data engineering services (AWS, GCP, Azure) to build scalable and cost-efficient data solutions
• Manage and maintain cloud-based data infrastructure

Data Modeling and Optimization:

• Design and implement data models to support business intelligence and analytics
• Optimize data structures and indexes for query performance

Collaboration and Communication:

• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and deliver actionable insights
• Effectively communicate complex technical concepts to both technical and non-technical audiences

Data Governance and Security:

• Implement data governance policies and procedures to ensure data quality, security, and compliance
• Protect sensitive data through appropriate security measures

Requirements:

• At least a bachelor’s degree in computer engineering, or equivalent
• At least 10 years in Microservices, with at least 4 years of experience in data engineering and architecture
• Previous work experiences in using major cloud platforms (AWS, Azure, GCP) and containerisation technologies (Docker, Kubernetes)
• Proficient in Python and Pyspark
• Proficient in SQL and database design
• Knowledge and understanding of data warehousing, data lakes and big data technologies
• Knowledge of data modeling and ETL/ELT processes
• Knowledge of PostgreSQL and Oracle
• Strong problem-solving and analytical skills
• Able to work independently and as part of a team
• Lead and manage the development team including assigning tasks and monitoring progress

Preferred Requirements:

• Experience in using data visualization tools e.g. Tableau, PowerBI