Data Engineer

About the Role

KPI Partners is looking for a skilled and driven Data Engineer to become part of our energetic team. This position involves designing, developing, and deploying scalable data solutions utilizing Microsoft Fabric technologies. The role is critical in crafting modern data platforms, efficient data pipelines, and empowering decision-making through data across the organization.

Key Responsibilities

• Design and develop data integration workflows using Microsoft Fabric to consolidate data flow from diverse enterprise sources.
• Partner with various teams to gather business requirements and convert them into technical design documents.
• Create and sustain scalable ETL/ELT pipelines to enable effective data extraction, transformation, and loading activities.
• Develop and enhance Fabric Data Factory Pipelines, Copy Activities, Dataflow Gen2 processes, and manage Notebook executions.
• Oversee full data workflow orchestration implementing the Bronze, Silver, and Gold layers in the Medallion Architecture.
• Set up pipeline scheduling, manage dependencies, implement retry strategies, handle timeouts, pass parameters, utilize variables, and craft dynamic expressions.
• Optimize pipeline operations focusing on performance, scalability, reliability, and maintaining data quality standards.
• Monitor pipeline processes, diagnose issues promptly, and ensure minimal disruption to business functions.
• Collaborate with data analysts, architects, and business stakeholders supporting analytics and reporting needs.
• Keep abreast of Microsoft Fabric advances and best practices in data engineering.

Qualifications

• Degree in Computer Science, Information Technology, Engineering, or a related discipline.
• 6 to 12 years of practical experience as a Data Engineer with substantial hands-on expertise in Microsoft Fabric.
• Proven experience designing and managing Data Factory Pipelines, Dataflow Gen2, and working with PySpark/Spark Notebooks.
• Strong command of SQL, data modeling techniques, and ETL/ELT pipeline development.
• Experience working with Lakehouse architecture and implementing Medallion Architecture comprising Bronze, Silver, and Gold layers.
• Familiarity with cloud-based data infrastructure and up-to-date data engineering methodologies.
• Excellent problem-solving, analytical, and troubleshooting skills.
• Effective communication and teamwork capabilities.
• Knowledge of Power BI or similar data visualization platforms is a plus.