Data Engineer

About the Role

We are looking for a seasoned Data Engineer to be a vital part of our elite data and analytics group which drives large-scale enterprise transformation projects. This position provides a chance to engage with cutting-edge cloud-native architectures, handle extensive datasets, and support AI applications through advanced data engineering technologies powering analytics, business intelligence, and machine learning solutions.

Primary Responsibilities

• Design and develop scalable data pipelines supporting both batch and streaming data processing.
• Construct and manage enterprise data lakes and lakehouse architectures, delivering refined analytical datasets.
• Write efficient SQL and Python code optimized for processing massive datasets.
• Implement comprehensive data quality controls, validation processes, monitoring tools, and observability systems.
• Create cloud-native data infrastructure solutions and leverage infrastructure-as-code for deployment automation.
• Work collaboratively with Data Scientists, BI teams, and business stakeholders to build dependable, governed data products.
• Facilitate AI and machine learning initiatives by preparing high-quality training datasets and supporting MLOps workflows.

Experience and Expertise Required

• More than five years of experience in data engineering roles.
• Proven ability in designing and scaling ETL/ELT data pipelines.
• Advanced proficiency in SQL and Python programming.
• Practical experience with cloud data platforms, specifically Microsoft Azure.
• Knowledge of modern data warehousing and lakehouse concepts.
• Familiarity with CI/CD pipelines and infrastructure automation best practices.

Essential Skills

• Microsoft Fabric
• Azure Data Factory (ADF)
• Databricks platform
• PySpark
• Python programming
• Advanced SQL querying
• Lakehouse architecture design
• Delta Lake technology
• Azure Synapse Analytics
• Terraform for infrastructure as code
• CI/CD using GitHub Actions and Azure DevOps
• Data governance and quality frameworks
• Change Data Capture (CDC) techniques
• API integration
• Real-time streaming data pipelines
• MLOps enablement
• Metadata and data lineage management

Preferred Additional Skills

• Knowledge of Scala programming
• Exposure to Hadoop ecosystem
• Experience in machine learning pipeline development
• Support for enterprise-wide AI projects

Additional Information

If you have a passion for tackling complex data engineering challenges and building reliable platforms that empower data-driven decision-making on a large scale, we encourage you to apply.