DataOps Consultant - Lakehouse Platform Support

Project Overview

This role involves providing Level 2 DataOps consultancy and support for Lakehouse data platforms within an investment management context. The consultant will focus on operational diagnostics, incident management, and ensuring stability across data pipelines encompassing batch and streaming data flows.

Primary Responsibilities

• Manage end-to-end L2 incident resolution for Lakehouse data platforms.
• Perform thorough operational diagnostics for both batch and streaming data pipelines.
• Collaborate closely with Platform Engineering, Data Engineering, and business teams to drive issue resolution.
• Use ITSM tools such as ServiceNow to handle incidents efficiently.
• Leverage metadata, data lineage, and available technical documentation to troubleshoot problems.
• Maintain clear ownership and communication, including escalation, during incident handling.
• Conduct root cause analysis (RCA) to identify and implement preventative measures.
• Support change management initiatives to maintain operational stability.

Essential Qualifications and Skills

• Proven experience in Data Operations with focus on Level 2 support.
• Hands-on expertise with Lakehouse architectures, particularly using Databricks, Delta Lake, or similar technologies.
• Strong command over incident management aligned with ITSM processes.
• Ability to troubleshoot batch and streaming data workflows.
• Knowledge of metadata-driven operations, data lineage tracking, and observability for data pipelines.
• Proficiency in Root Cause Analysis, change and problem management methodologies.
• Experience taking ownership of incidents through their lifecycle, from identification to resolution.
• Familiarity with investment management data domains including portfolio and positions, trade lifecycle, market data, and risk & performance metrics.
• Background as a Data Engineer working with Databricks and Delta Lake, including a technical understanding of Delta Lake internals.

Desirable Skills

• Experience with Palantir Foundry, especially data quality control setups and pipeline diagnostics.
• Knowledge of Monte Carlo tools for data quality monitoring and observability.
• Familiarity with Azure DevOps for repository management and continuous integration/continuous deployment (CI/CD) pipelines.
• Exposure to enterprise-level observability tools like Dynatrace.
• Understanding of batch orchestration and workflow management systems.
• Awareness of investment management platforms such as Optima and Cardia.
• Strong grasp of portfolio concepts such as maturity date and value date.